Inside SitSense: A Browser-Only AI Posture Coach

• The browser uses @mediapipe/pose and a face landmarker to estimate landmarks.
• TypeScript utilities convert those landmarks into seven ergonomic metrics and a posture score.
• React renders a live coach view with scores, streaks, and points. Frames never leave the tab.
• Every few seconds we send structured posture metrics to a Postgres and Supabase backend, then use an LLM to summarize the session.

1. 01

   Requests webcam access through getUserMedia and attaches the stream to a hidden video element.

2. 02

   Runs MediaPipe Pose on incoming frames to get a 2D body skeleton with 33 landmarks.

3. 03

   Runs a lightweight face landmarker at a lower frame rate to estimate 3D head pose.

4. 04

   Sends pose and face data into calculatePostureMetrics to compute seven angles and offsets.

5. 05

   Updates the on screen posture score, streak timers, and points in React, while throttling metric saves to the backend.

Anything that changes every frame lives in refs, not state. This keeps React from rendering on every camera frame and lets the UI feel responsive without turning the component into a frame loop.

• Neck angle relative to vertical.
• Craniovertebral angle (CVA), a standard head–neck alignment measure.
• Trunk angle to capture torso lean.
• Shoulder slope, which shows side loading.
• Head forward relative to the torso.
• Head tilt or roll.
• Neck turn or yaw.

• Warmup frames ignore the first set of samples for each metric. During this period the UI stays neutral rather than pretending to know your baseline.
• Rolling history buffers keep around thirty seconds of data for key metrics at a reduced frame rate. Baselines blend medians with exponential moving averages.
• Calibration aware classification hides or softens scores until the system is confident that a metric has settled.

The goal is simple. SitSense should say you are drifting forward relative to your own neutral, not relative to a model neck in a textbook.

• Current posture state such as good or fair or poor.
• How long you have stayed in good posture.
• When and how you recover after a slouch.

POST /api/posture/save
{
  session_id: "uuid",
  user_id: "uuid",
  metrics: {
    neck_angle: number,
    cva_angle: number,
    trunk_angle: number,
    shoulder_slope: number,
    head_forward: number,
    head_tilt: number,
    neck_turn: number
  },
  posture: "good" | "fair" | "poor",
  confidence: number
}

The server validates this payload with Zod and stores it in a posture_analysis table in Postgres through Supabase and Drizzle. Writes are rate limited per user so that a long session generates a reasonable number of rows instead of a full frame log.

1. 01

   Fetch all posture rows for a session from posture_analysis.

2. 02

   Bucket samples into five second windows and pick the worst metric in each bucket.

3. 03

   Map each combination of metric and severity to a letter in a limited alphabet.

4. 04

   Build a strict prompt that includes the mapping but forbids the model from echoing the code letters.

5. 05

   Ask the LLM for a short, plain text summary and store it alongside the aggregates in a sessions.analysis field.

The LLM never sees individual metric values or timestamps, only the compressed sequence and session length. The result feels human and readable without giving up the privacy story.

• Client: Next.js App Router, React, TypeScript, @mediapipe/pose, @mediapipe/tasks-vision.
• State discipline: per frame values in refs, state updates only when the UI needs to change.
• Backend: Next.js API routes, Supabase, Postgres, Drizzle ORM, Zod for validation.
• Analysis: Gemini models behind a strict prompt, with retries and model fallbacks.

The same architecture could support other ergonomic use cases such as lifting form checks or standing desk posture without changing the privacy story. You would update the metrics and thresholds, not the basic flow.