An eye-gaze controlled Augmentative and Alternative Communication (AAC) platform with smart phrase expansion for ALS patients.
Eyra is a browser-based assistive typing environment. Through browser-native webcam gaze tracking, dwell-time keyboard selection, and Gemini-powered smart shorthand expansion, it offers a private, accessible, and low-fatigue space for independent communication.
- Overview
- Features
- Product Experience
- Architecture
- Technology Stack
- Design Philosophy
- Local Development
- Future Roadmap
- Team
- License
Eyra was built for individuals with severe motor and speech impairments (such as ALS or motor neuron diseases). By leveraging standard webcams and on-device machine learning, Eyra eliminates the need for expensive, proprietary eye-tracking hardware. Users calibrate their webcam, type via eye-dwell on an optimized keyboard layout, and expand abbreviated shorthands into natural, spoken sentences instantly.
- No-Hardware Eye Gaze Tracking — Track gaze points using a standard webcam. Supports two modular, browser-based engines:
- MediaPipe Face Landmarker — High-precision facial landmark mesh tracking.
- WebGazer.js — Self-calibrating mapping model directly in the browser.
- Visual Gaze Calibration — A 9-point interactive setup screen to fine-tune accuracy to the user's specific posture, webcam angle, and lighting conditions.
- GazeBoard Keyboard — An on-screen typing board custom-designed for eye movement. Features customizable layout actions and a text-to-speech option to vocalize typed statements.
- Custom Dwell Time Engine — Built on a custom
useDwellReact hook that tracks continuous gaze targets, coordinates selection thresholds, and visualizes selection progress via radial loaders. - Smart Phrase Expansion — Instantly expands short, eye-typed abbreviations (e.g.,
BTHMto"I need to use the bathroom."orWATto"I would like some water please."):- Explicit fallback — If Gemini is unavailable, Eyra reports that clearly or uses a constrained safety rule; it never presents a guessed sentence as model output.
- Gemini — Context-aware AI engine to predict and build full, natural statements.
- Express & MCP Servers — Backend integrations supporting both standard REST API calls and the Model Context Protocol (MCP) to plug Eyra directly into larger LLM workflows.
- Independent Testing Suite — A standalone sandbox environment under
/testingfeaturing clean Vite configurations to test gaze engine performance in isolation. - Sustainable Development Goals (SDG) — Aligned with UN SDGs (Good Health & Well-being, Quality Education, Reduced Inequalities, Partnerships).
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ 1. Calibrate │ ───> │ 2. Look to Type │ ───> │ 3. Smart Expand │
└──────────────────┘ └──────────────────┘ └──────────────────┘
▲ │
│ ▼
┌──────────────────┐ ┌──────────────────┐
│ 5. SDG & Research│ <───────────────────────────── │ 4. Speak & Send │
└──────────────────┘ └──────────────────┘
- Calibrate — Complete the quick 9-point gaze calibration to map screen coordinates to your eyes.
- Look to Type — Focus your gaze on keys to type letters or common words on GazeBoard.
- Smart Expand — Request AI expansion to transform your short typed letters into natural sentences.
- Speak & Send — Use built-in Text-To-Speech (TTS) to vocalize your sentence.
- Research & SDG Impact — Review SDG alignments, system architectures, and responsible AI documentation.
graph TD
classDef browser fill:#fbfff4,stroke:#a2bfa7,stroke-width:1px,color:#203727;
classDef processing fill:#fbfff4,stroke:#3E8B5C,stroke-width:1px,color:#203727;
classDef data fill:#f5fbf6,stroke:#7a9080,stroke-width:1px,color:#203727;
A[Webcam / Video Input]:::browser -->|Gaze Points| B[Gaze Tracking Engines]:::browser
B -->|WebGazer.js / MediaPipe| C[Custom useDwell Hook]:::browser
C -->|Dwell Time Trigger| D[GazeBoard Keyboard]:::browser
D -->|Abbreviated Input| E[AppPage State]:::browser
E -->|Express API Request| F[API Server]:::processing
E -->|MCP Tool Request| G[MCP Server]:::processing
F -->|Gemini AI expansion| H[Gemini 3.5 Flash]:::processing
G -->|Gemini AI expansion| H
F -->|Explicit rule/error fallback| I[Safety Fallback]:::processing
H -->|Expanded Sentence| E
I -->|Transparent fallback| E
E -->|Web Speech API| J[Text-to-Speech Output]:::browser
D -->|Paired patient request| K[Supabase care_requests]:::data
K -->|Realtime / authenticated polling| L[Caregiver Web + Android]:::browser
K -->|Database trigger| M[notification_outbox]:::data
M -->|Webhook| N[notify-push Edge Function]:::processing
N -->|Server-only API key| O[OneSignal]:::processing
L -->|Authenticated reply| P[Supabase caregiver_messages]:::data
P -->|Realtime + read receipt| E
| Technology | Version | Purpose |
|---|---|---|
| Vite | 8.1.1 | High-speed frontend build tooling |
| React | 19.2.7 | UI layout and reactive views |
| TypeScript | 6.0.2 | Static typing across components and servers |
| Tailwind CSS | 4.3.2 | CSS utility tokens and component styling |
| Framer Motion | 12.42.2 | Micro-animations and page transitions |
| MediaPipe Vision | 0.10.35 | On-device face and gaze landmark mesh detection |
| Gemini 3.5 Flash | — | Smart phrase expansion and context building |
| MCP SDK | 1.29.0 | Model Context Protocol integration |
| Express | 4.22.2 | REST API server backend |
| Supabase | — | Authenticated caregiver pairing, Postgres RPC, RLS, and realtime data |
| Kotlin / Jetpack Compose | — | Native caregiver companion app |
| OneSignal | — | Server-triggered caregiver push notifications |
| Oxlint | 1.71.0 | Fast, efficient code linting |
Eyra incorporates visual design choices designed specifically for accessible user experiences.
- Contrast & Calm — Dominated by a soft cream background (
#fbfff4) and deep forest green (#203727) to limit screen eye strain. - Fatigue Reduction — Large button targets and visual feedback rings prevent "Midas Touch" (accidental typing activations) while maintaining high speed.
- Autonomy & Access — Simple webcam-only access means anyone can communicate independently without costly, proprietary eye-tracking devices.
- Node.js 18+
- npm
# Clone the repository
git clone https://github.com/arj-co/Eyra.git
cd Eyra
# Install dependencies
npm installCreate a .env file in the project root:
# Gemini API Key (for smart phrase expansion)
GEMINI_API_KEY=your_gemini_api_key_here# Start the Vite React app
npm run dev
# Start the Express API server (runs on port 5001)
npm run server
# Start the MCP Server (runs over stdio transport)
npm run mcp
# Run the static checks used before a demo handoff
npm run verify:demo
# Deploy the notification worker after linking Supabase and setting secrets
npm run deploy:notify-pushEyra includes an opt-in local AI expansion mode available at /local.
- Optional: Ollama is not required to run Eyra. If Ollama is absent, the rest of Eyra (Gemini expansion, GazeBoard, caregiver sync) works normally.
- No Cloud API Key Required: Local Ollama runs completely on your machine and does not need any cloud API keys.
- Local Machine Requirement: Ollama must be installed and running on the same device as your browser (
http://127.0.0.1:11434). - One-Time Model Download: A model (such as
qwen2.5:1.5b) must be pulled once onto your machine before generating local phrases. - Scope Notice: Local mode provides on-device LLM text expansion. It does not automatically provide offline eye tracking, caregiver delivery, push notifications, or full end-to-end encryption.
# Download a lightweight local model
ollama pull qwen2.5:1.5b
# Start the local Ollama service (if not already running as a desktop app)
ollama serveNavigate to /local in your browser to select your installed model and expand phrases privately on your device.
- Multi-lingual Expansions — Support for regional Indic languages (Hindi, Kannada, Telugu).
- Custom Key Layouts — Support for ABC, frequency-based, or customizable key mappings.
- On-Device LLMs — Smart expansion using local, lightweight on-device models.
Eyra pairs a patient device with a caregiver account through a short-lived code:
- The caregiver signs in to Eyra Care (web or Android) and creates a one-time code.
- The patient enters that code in Eyra; the patient device claims the code and is linked to the caregiver.
- The caregiver dashboard reads the authenticated account's connected patients from Supabase and refreshes automatically while keeping a local cache for offline viewing.
The Android caregiver companion lives in eyra-caregiver-android. It uses the public Supabase key only, Supabase Auth for the caregiver session, Row Level Security for patient access, and OneSignal for push delivery. The Supabase service key is never included in the web or Android clients.
The demo path is backed by real data rather than a local mock:
- A paired patient selection inserts a row into
care_requeststhrough the patient Supabase session. - The caregiver web dashboard or Android companion reads that row and can acknowledge it.
- A caregiver reply is inserted into
caregiver_messages; the patient board receives it through Supabase Realtime and can mark it read. - If the patient loses connectivity, the request stays visibly queued in browser storage and retries after the browser reports it is online. It is not shown as delivered until Supabase accepts it.
- The database trigger writes a notification to
notification_outbox. To enable actual OneSignal delivery, deploysupabase/functions/notify-pushand configure theONESIGNAL_APP_ID,ONESIGNAL_REST_API_KEY, and webhook secret in Supabase. Realtime remains the source of truth until that server-side push step is configured.
Apply supabase/migrations/20260805_caregiver_messaging_and_push.sql before running the reply demo. The migration creates the message table, RLS policies, realtime publication, and notification outbox; no client receives a service-role credential.
The deployed demo uses the pg_net trigger in supabase/sql/notify-push-webhook.sql to call the Edge Function after each outbox insert. Replace the two placeholders with the project ref and the server-only webhook secret in the Supabase SQL editor. The trigger is deliberately kept separate from the migration so a webhook secret can never enter git history.
Camera frames and gaze calibration stay in the patient browser. Supabase publishable keys are protected by Row Level Security. Caregiver replies and request events are stored only for the paired care circle. OneSignal API keys and the Supabase service role key belong only in Edge Function secrets; they are never bundled into the web app or Android APK.
- Arjun — AI & Gaze Tracking
- Shrikar — Frontend & AAC UX
- Harsh — Backend & Integration
- Srushti — Research & QA
This project is licensed under the terms of the MIT License.