Build log · Software & apps

GESTO 2000 — a webcam gesture toy rebuilt to run with no CDN

Software & appsdone1 versionsMediaPipezero CDN42-dim hand featuresno-server duel18 selftests
📄 Raw spec (.md)📁 Portfolio

reflection not written yet · every fact below is generated from this project’s own spec and files

What it is

A rebuild of gestomeme.com: the Windows 98 desktop where you teach a webcam your own hand gestures, mint them into reaction cards, sweep mines and duel a friend. The premise is kept and the reason it dies on a school connection is gone — the 21 MB of MediaPipe models and WASM ship with the page instead of being fetched from storage.googleapis.com, which is blocked in mainland China and slow everywhere else. Per-gesture tolerances measured from your own jitter, 42-dimension hand features that survive distance and laterality, 52 blendshapes for faces, three languages, and a duel that exchanges one pasted code each way with no server at all. 18 selftest checks green, verified by driving the real page with a fake camera.

Version history

Straight from the project’s own card data — what each version added.

V1

140,498-byte shell · 21 MB of models served same-origin · per-gesture tolerance · 3 languages · SELFTEST 18/18 · graded at 1440 and 390

Files on the site

What broke, and what I’d change

Not written yet. This is the part only I can write — what actually went wrong, and what I’d do differently. It lives in builds/reflections/gesto.md (see _TEMPLATE.md); as soon as that file has words in it, this section fills in on the next build.

Everything else on this page — the spec, the version history, the file counts — is generated from the project’s own build notes and files, so nothing here is guesswork.

Build spec

The project’s own build notes, rendered. Open the raw .md ↗

Show the spec

Build spec — GESTO 2000 (webcam gesture toy, no CDN)

Goal

A rebuilt, improved version of gestomeme.com: a retro desktop where you teach a webcam your own hand gestures, mint them into reaction cards, sweep mines and duel a friend. Keep the premise and the look; remove the thing that makes the original unusable on a Chinese or school network.

The load-bearing constraint

The original loads its MediaPipe bundles and .task models from storage.googleapis.com — blocked in mainland China, slow everywhere else — and fails silently: the loading line never resolves.

| Requirement | Implementation |
|---|---|
| No third-party origin for app assets | vendor/vision_bundle.mjs, wasm/, models/.task fetched with relative URLs from the app's own directory |
| No framework, no build server | one output index.html, produced by build.py concatenating src/* |
| Recognises hands and faces | hand_landmarker.task (21 landmarks) + face_landmarker.task (478 landmarks, 52 blendshapes) |
| Works at 390 px wide | real tab bar below 760 px, no fixed-pixel panels, zero horizontal overflow |
| Three languages, no English-only fallback | every string is a [繁, 简, EN] triple in 30_i18n.js |

Recognition

  • Hand features (42 dims) — 21 landmarks, wrist-origin, divided by palm size (wrist → middle

    • knuckle), x and y only. Scored by L1 distance against the template and its mirror, so laterality is free and distance from the lens is irrelevant.

  • Face features (52 dims) — blendshape coefficients, which are already distance-invariant.
  • Template — per-dimension median of accepted frames; a frame is accepted only if the detector's

    • hand count matches the recording's, so a stray second hand cannot poison the template.

  • Tolerance — clamp(0.075 + 3.6 × MAD, 0.09, 0.30) per gesture. Replaces the single global

    • threshold that made every gesture either trigger-happy or deaf.

  • Fire rule — best match must win 4 consecutive frames, then a 1.6 s cooldown.
  • Face pass budget — ~11 Hz, previous result reused between detections (the face model costs ~3×

    • the hand model; running it at 30 Hz put software rendering at 2 fps).

Duel

No signalling server. Offer = JSON → CompressionStream('gzip') → base64, pasted by hand; the answer comes back the same way; then RTCPeerConnection + datachannel, peer to peer. Documented in-window in all three languages so the manual exchange is not a mystery.

Window manager

Draggable, resizable, minimise/maximise, z-order, taskbar, cascade. POS holds the intended geometry per window; clampIntoStage() must ignore closed windows or their inline left/top gets rewritten to 0px and show() stops parking them.

Acceptance (all must pass before publish)

  1. python3 build.py emits index.html and prints its sha256.
  2. #selftest reports 18/18: i18n completeness (every key has three values), matcher, duel codec,

    3. a real RTCPeerConnection datachannel loopback, minesweeper placement/cascade/win, window manager, card rendering.

  3. Headless Chrome with --use-file-for-fake-video-capture: hand detected → gesture taught at 42 dims →

    4. live score ≥ 0.9 → card captured.

  4. 1440×900 and 390×844: no console errors, no horizontal overflow, no clipped panels.
  5. Zero requests to any host other than the app's own origin (checked in the network panel).