Colombian Sign Language interpreter with computer vision and LLMs.
Problem & solution
Deaf people face huge barriers to communicate in digital environments: there are almost no tools that interpret sign language fluently. This project — built by a team of 4 for Continuous Systems Modeling & Simulation (Universidad de Pamplona) — proposes and models a layered intelligent system: a perception layer extracts the 21 hand landmarks in real time with MediaPipe from the webcam; intelligent agents (i5.0 architecture) translate motion vectors into information tokens; and a Large Language Model structures those signs into grammatically correct, contextualized Spanish sentences, adapted to Colombian Sign Language (LSC). The prototype runs directly in the browser and is deployed in production.
Highlights
- Real-time computer vision
- i5.0 agents + LLM
- Accessibility & inclusion
- In production
What it does
- [01]
Real-time hand detection from the webcam: extraction of the 21 landmarks with MediaPipe.
- [02]
Intelligent agents (i5.0 architecture) that turn motion vectors into information tokens.
- [03]
LLM that interprets the tokens and builds grammatically correct, context-aware Spanish sentences.
- [04]
Adapted to Colombian Sign Language (LSC), with a regional inclusion focus.
- [05]
Web prototype accessible from the browser, deployed in production.
- [06]
Complete academic documentation: state of the art, layered architecture and validation.
Try the app
The demo opens in a new tab.
Screenshots
Click a screenshot to enlarge it
Tech stack
- Python
- MediaPipe
- Visión Computacional
- LLM
- Agentes IA
- JavaScript