You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
flowchart LR
subgraph Browser
UI[Next.js App]
ONNX[ONNX Runtime]
IDB[(IndexedDB)]
end
subgraph AWS["AWS Cloud"]
AMP[Amplify SSR]
BED[Bedrock]
POL[Polly]
DDB[(DynamoDB)]
end
UI -->|SSR + API| AMP
AMP -->|Nova Lite/Pro| BED
AMP -->|Neural TTS| POL
AMP -->|CRUD| DDB
UI -->|Inference| ONNX
UI -->|Local data| IDB
AMP -->|Static assets| S3[S3 Bucket]
classDef aws fill:#FF9900,color:#232F3E,stroke:#232F3E
classDef browser fill:#4A90D9,color:#fff,stroke:#2C5F8A
classDef storage fill:#3B48CC,color:#fff,stroke:#2C3599
class AMP,BED,POL,S3 aws
class UI,ONNX browser
class DDB,IDB storage
Loading
Key principle: The core screening AI (pose detection + behavior classification) runs entirely on-device via ONNX Runtime Web. AWS services handle generative AI enrichment, authentication, data sync, and hosting — all with graceful fallbacks so the app works even when cloud services are unavailable.
Service Inventory
#
Service
Resource
Region
Status
Cost Tier
1
Amazon Bedrock
Nova Lite v1 + Nova Pro v1
us-east-1
Active
Pay-per-token
2
Amazon Polly
Neural TTS (Joanna)
ap-south-1
Active
Pay-per-char
3
Amazon DynamoDB
7 tables (5 active, 2 placeholder)
ap-south-1
Active
On-demand
4
Amazon S3
autisense-models-762099405044
ap-south-1
Configured
Standard
5
AWS Amplify
WEB_COMPUTE SSR
ap-south-1
Active
Build + hosting
6
AWS IAM
1 user + 1 role + policies
Global
Active
Free
7
AWS Budgets
$10/month alarm
Global
Active
Free
Services NOT used: EC2, ECS, API Gateway (Amplify handles routing internally), SageMaker, Kiro.
Amazon Bedrock (Generative AI)
Models
Model ID
Model Name
Purpose
Routes
Inference Config
amazon.nova-lite-v1:0
Nova Lite
Chat, words, summary
3 routes
maxTokens: 256-1024, temp: 0.7-0.8
amazon.nova-pro-v1:0
Nova Pro
Clinical reports
1 route
maxTokens: 512, temp: 0.5
Both models accessed via BedrockRuntimeClient + InvokeModelCommand from @aws-sdk/client-bedrock-runtime.
Route Details
1. AI Voice Conversation — /api/chat/conversation
File: app/api/chat/conversation/route.ts
Model: Nova Lite
Purpose: Adaptive multi-turn conversation with child during screening (social, cognitive, language, motor domains)
Prompt: System instructions embedded in first user message (Nova Lite has no system role). Multi-turn history maintained.
Response: JSON extraction from markdown code blocks or raw text
Timeout: 3-second AbortController on SDK call
Fallback: Pre-defined conversation pool (greetings, mid-questions, farewells)
flowchart TD
REQ[Client Request] --> ROUTE[API Route Handler]
ROUTE --> BUILD[Build Prompt]
BUILD --> SDK[BedrockRuntimeClient]
SDK --> MODEL{Nova Lite or Pro}
MODEL -->|Success| PARSE[Parse JSON Response]
MODEL -->|Timeout/Error| FALL[Fallback Response]
PARSE --> RES[Return to Client]
FALL --> RES
classDef aws fill:#FF9900,color:#232F3E
classDef logic fill:#34A853,color:#fff
classDef err fill:#EA4335,color:#fff
class SDK,MODEL aws
class ROUTE,BUILD,PARSE,RES logic
class FALL err
Loading
Fallback Behavior
Route
Trigger
Fallback Strategy
/api/chat/conversation
Timeout (3s) or SDK error
Pre-defined conversation pool (greetings, mid-questions, farewells)
/api/chat/generate-words
Timeout (3s) or SDK error
Age-stratified curated word/sentence pools with emojis
/api/report/summary
Any Bedrock failure
Template mock summary with placeholder insights
/api/report/clinical
Any Bedrock failure
Deterministic template-only report (no AI enrichment)
Amazon Polly (Text-to-Speech)
Setting
Value
Route
POST /api/tts
File
app/api/tts/route.ts
SDK
PollyClient + SynthesizeSpeechCommand
Default Voice
Joanna
Engine
Neural
Output Format
MP3
Max Text
3000 chars (truncated at 2900 at sentence boundary)
Region
POLLY_REGION env var or ap-south-1
Cache
Cache-Control: public, max-age=86400
Error
Returns HTTP 503
Client-side fallback: When Polly fails, the client (app/lib/tts/ttsHelper.ts) falls back to browser speechSynthesis API with rate=0.8, pitch=1.1.
Used by: Word Echo challenge (Step 4), Speech Practice game, AI conversation voice playback.
Amazon DynamoDB (Data Layer)
Tables
#
Table Name
PK
SK
GSI
TTL
Status
Purpose
1
autisense-users
id (S)
—
email-index (email → id)
—
Active
User accounts (Google OAuth)
2
autisense-auth-sessions
token (S)
—
—
expiresAt
Active
Session tokens (24h expiry)
3
autisense-sessions
id (S)
—
—
ttl (365d)
Active
Anonymized screening sessions
4
autisense-biomarkers
sessionId (S)
—
—
ttl (365d)
Active
Aggregated biomarker data
5
autisense-feed-posts
postId (S)
createdAt (N)
—
—
Active
Community feed posts + reactions
6
autisense-child-profiles
—
—
—
—
Placeholder
Future child profile storage
7
autisense-session-summaries
—
—
—
—
Placeholder
Future session summary cache
Authentication Flow
sequenceDiagram
participant B as Browser
participant A as Amplify SSR
participant G as Google OAuth
participant D as DynamoDB
B->>A: GET /api/auth/google
A->>G: Redirect to Google
G->>A: Callback with auth code
A->>G: Exchange code for tokens
G-->>A: User info (email, name)
A->>D: upsertGoogleUser()
D-->>A: User record
A->>D: createAuthSession()
D-->>A: Session token
A->>B: Set cookie + redirect
B->>A: GET /api/auth/session
A->>D: getAuthSession(token)
D-->>A: Session + User
A->>B: User JSON
Loading
Auth Library — app/lib/auth/dynamodb.ts
Operation
DynamoDB Command
Table
createUser()
PutCommand
users
getUserByEmail()
QueryCommand (GSI)
users
getUserById()
GetCommand
users
updateUser()
UpdateCommand
users
createAuthSession()
PutCommand
auth-sessions
getAuthSession()
GetCommand + expiry check
auth-sessions
deleteAuthSession()
DeleteCommand
auth-sessions
upsertGoogleUser()
Query + conditional Put
users
Data Sync — app/api/sync/route.ts
Field
Detail
Trigger
Client calls POST /api/sync after screening completes
How it works: Amplify builds the Next.js app, deploys static assets to CloudFront CDN, and runs SSR pages + API routes as Lambda functions behind an internal API Gateway. No explicit Lambda or API Gateway configuration needed — Amplify manages this automatically.
Amazon S3 (Object Storage)
Setting
Value
Bucket
autisense-models-762099405044
Region
ap-south-1
SDK
Removed (@aws-sdk/client-s3 and @aws-sdk/s3-request-presigner uninstalled — zero usage in code)
Status
Bucket exists for backup. Not used at runtime — models served from public/ via CDN
Current approach: ONNX models (YOLO26n, BodyTCN, FaceTCN, FER+) are served from public/models/ as static assets via Amplify's CloudFront CDN. This is simpler and faster for the 4 small models (~5-15 MB each). S3 bucket exists as a future option for larger model storage or versioned model management.
AWS IAM (Access Control)
Resource
Type
Purpose
autisense-app
IAM User
Programmatic access for APP_* credentials
AutiSenseAmplifyRole
IAM Role
Amplify build + Lambda execution role
AutiSenseBedrockPolicy
IAM Policy
Allows bedrock:InvokeModel for Nova Lite + Nova Pro
Default chain
SDK Provider
Lambda runtime inherits role credentials automatically
Policy scope: AutiSenseBedrockPolicy is scoped to amazon.nova-lite-v1:0 and amazon.nova-pro-v1:0 only. Cohere models removed from policy after migration to Nova.
Priority 1: SDK default credential chain
└─ Covers: Lambda IAM roles (Amplify), local dev with AWS_* env vars
Priority 2: Custom APP_* environment variables
└─ Covers: Amplify deployments (AWS_* prefix is reserved/blocked by Amplify)
Function
Returns
Used By
getAppCredentials()
{ accessKeyId, secretAccessKey } from APP_* vars, or undefined
All Bedrock, Polly, DynamoDB clients
getAppRegion(fallback)
APP_REGION → AWS_REGION → fallback
All AWS SDK clients
Why APP_ instead of AWS_**: Amplify reserves the AWS_* environment variable prefix. Setting AWS_ACCESS_KEY_ID in Amplify console is blocked. The workaround uses APP_ACCESS_KEY_ID and APP_SECRET_ACCESS_KEY instead.
API Route Matrix
#
Route
Method
AWS Services
Fallback
File
1
/api/auth/google
GET
None
—
app/api/auth/google/route.ts
2
/api/auth/callback/google
GET
DynamoDB
In-memory auth
app/api/auth/callback/google/route.ts
3
/api/auth/session
GET
DynamoDB
In-memory auth
app/api/auth/session/route.ts
4
/api/auth/logout
POST
DynamoDB
In-memory auth
app/api/auth/logout/route.ts
5
/api/chat/conversation
POST
Bedrock (Nova Lite)
Conversation pool
app/api/chat/conversation/route.ts
6
/api/chat/generate-words
POST
Bedrock (Nova Lite)
Curated word pools
app/api/chat/generate-words/route.ts
7
/api/feed
GET/POST
DynamoDB
In-memory (30s)
app/api/feed/route.ts
8
/api/nearby
POST
None (Overpass API)
—
app/api/nearby/route.ts
9
/api/report/clinical
POST
Bedrock (Nova Pro)
Template-only
app/api/report/clinical/route.ts
10
/api/report/pdf
POST
None (pdf-lib)
—
app/api/report/pdf/route.ts
11
/api/report/summary
POST
Bedrock (Nova Lite)
Mock summary
app/api/report/summary/route.ts
12
/api/report/weekly
POST
None (IndexedDB)
—
app/api/report/weekly/route.ts
13
/api/sync
POST
DynamoDB
Error response
app/api/sync/route.ts
14
/api/tts
POST
Polly
503 (client uses speechSynthesis)
app/api/tts/route.ts
Summary: 14 API routes total. 10 use AWS services (4 Bedrock, 1 Polly, 5 DynamoDB). 4 routes have no AWS dependency.
Standalone Lambda Handler
File
Purpose
AWS Services
server/lambda/sync-handler.ts
Alternative async sync (API Gateway → Lambda)
DynamoDB (sessions + biomarkers)
This handler is a direct Lambda function (not Next.js API route) that accepts APIGatewayProxyEventV2 events. It mirrors /api/sync logic but runs independently. Currently available but not the primary sync path — the Next.js API route handles sync in the Amplify deployment.
Screening Data Flow
flowchart TD
CAM[Webcam Feed] --> YOLO[YOLO26n Pose]
YOLO --> FEAT[Feature Encoder]
FEAT --> BTCN[BodyTCN]
CAM --> FACE[Face ROI Crop]
FACE --> FER[FER+ Emotions]
FACE --> MP[MediaPipe Landmarks]
FER --> FENC[Face Encoder]
MP --> FENC
FENC --> FTCN[FaceTCN]
BTCN --> FUS[Late Fusion 70/30]
FTCN --> FUS
FUS --> BIO[Biomarker Scores]
BIO --> IDB[(IndexedDB)]
IDB -->|POST /api/sync| DDB[(DynamoDB)]
DDB -->|POST /api/report| BED[Bedrock Nova]
BED --> PDF[PDF Report]
classDef device fill:#4A90D9,color:#fff
classDef aws fill:#FF9900,color:#232F3E
classDef storage fill:#3B48CC,color:#fff
class CAM,YOLO,FEAT,BTCN,FACE,FER,MP,FENC,FTCN,FUS,BIO device
class BED,PDF aws
class IDB,DDB storage
Loading
On-device (blue): Everything from webcam capture through biomarker scoring runs in the browser via Web Workers + ONNX Runtime. Zero cloud dependency for core screening.
Cloud (orange): Only report generation uses AWS — biomarkers are synced to DynamoDB, then Bedrock enriches the clinical report. The screening works fully offline; cloud adds clinical depth.
On-Device AI (No Cloud)
The core screening pipeline runs entirely in the browser using ONNX Runtime Web. No data leaves the device during inference.