Promote Sign in Sign up

Repository

kid-mode-jarvis

A JARVIS-style Gemini Live voice console for kids - server-authoritative and safety-walled. The AI can never award a real-world achievement; only a parent can.

Hermes815
Repository

README

Kid Mode — a JARVIS for your kid

A real-time voice + screen AI console for a young child, built on
google-gemini/live-api-web-console.
It boots a tablet straight into an Iron-Man-style JARVIS who teaches, runs an earned
rank ladder, controls the house music, and can place a real call to Dad — while a
Node server, not the language model, holds authority over everything that has to be true.

The one idea that makes it safe: Gemini performs the character. The server controls reality.
The voice model can improvise a mission, tell a story, or turn a lesson into an experiment.
It cannot decide whether the child answered correctly, award a rank, mark a physical
project complete, claim a song started, or invent a real-world fact. Rope on actions, rails on truth.

This is the surface a child actually uses: a locked-down tablet with no browser, no app drawer,
no YouTube — just the console.

What it does

  • Native Gemini Live conversation with a British-RP, film-JARVIS voice; barge-in + deliberate mute.
  • Grounded answers only. Any real-world factual question is routed to a grounding LLM (see below)
    or gets an in-character "that's a question for your dad" — the voice model is never a source of truth.
  • Seven-rank service ladder (Cadet → Chief Engineer) graded server-side from an authored
    curriculum. Ranks unlock real capabilities (house music, calling Dad). The top rank is gated on a
    physical build, not trivia — a probabilistic transcript can't declare a real object exists.
  • Learn / Do / Build: exploration bays, hands-on experiments and field missions, and sparring rounds.
  • Dad Link: a real two-way WebRTC voice call to a parent, wrapped in a "satellite search" theater.
  • Dad-authored dossier: true, rank-gated files about the child's father, revealed as a reward.
  • Approval-gated memory: the model can queue a literal quote/moment; a parent reviews before
    anything is kept. A separate Dad brief collects milestones/worries/funny moments.
  • Whole-house music via Home Assistant (optional), with deterministic start/stop the model can't fake.
  • Child-safety + adult-privacy walls: a categorical safety system-prompt plus server-side denylists.

Architecture — one engine, two halves

  Tablet browser (HTTPS, secure context)              Cloud / your host
  ┌──────────────────────────────┐                    ┌────────────────────────┐
  │  React app (this repo)        │   WebSocket        │  Google Gemini Live    │
  │   • Gemini Live voice         │◄──────────────────►│  (the PERFORMER)       │
  │   • JARVIS persona + tools    │                    └────────────────────────┘
  │   • on-screen cards           │
  └──────────────┬───────────────┘                    ┌────────────────────────┐
                 │ same-origin /api/*      Bearer key  │  Grounding LLM gateway │
                 ▼                        ┌────────────►│  (OpenAI-compatible    │
  ┌──────────────────────────────┐       │             │   /v1/chat/completions)│
  │  server/index.js (Node)       │──────┘             │  = the source of TRUTH │
  │   • serves the app over HTTPS │                    └────────────────────────┘
  │   • holds ALL secrets/keys    │
  │   • grades ranks, persists    │                    ┌────────────────────────┐
  │     state, enforces lockouts  │──── optional ─────►│  Home Assistant (music)│
  │   • proxies grounded answers  │                    └────────────────────────┘
  └──────────────────────────────┘

Why the server exists: the browser must not hold API keys, getUserMedia needs a secure
(HTTPS) context, and every truth-bearing decision must live somewhere the model can't reach. The
Node server does all four — key isolation, HTTPS serving, durable server-graded state, and a proxy
to the grounding LLM.

⚠ The mic gotcha — read this first

Browser voice input (getUserMedia) works only in a secure context — HTTPS or localhost.
Serving the app over a plain LAN IP silently blocks the mic: the app loads, the voice just never
hears anything. You must serve it over HTTPS. The easy path is a Tailscale
cert for your host (tailscale cert <host>.<tailnet>.ts.net); any real HTTPS cert works too. This
bites every browser-voice app, not just this one.

Requirements

  • A tablet — and the model matters more than you'd think. Don't buy on price alone; buy on
    these four capabilities, because a tablet that misses any of them fails silently (the console
    loads, the voice just never works):

    • A current Chromium/Android WebView — Gemini Live is a modern-web workload.
    • getUserMedia in a secure context — see the mic gotcha above. This is the usual killer.
    • WebRTC, if you want Dad Link's voice call.
    • A powered mount you intend to leave plugged in — Android power-saving throttles a
      discharging tablet until the whole system looks broken (slow, glitchy, drops Wi-Fi,
      vanishes from the network). It has cost a full debugging session. It was the charger.

    Tested: Samsung Galaxy Tab A9+ (SM-X210, ~$250). Recommended: its successor, the
    Galaxy Tab A11+. Evaluated and rejected: cheap Fire HD tablets — the ~$60 wall-tablet
    plan is tempting and it's where the mic dies. Any tablet meeting the four points above should
    work; the two Samsungs are simply the ones with real hours on them.

  • Fully Kiosk Browser for lockdown — the PLUS license
    (~one-time $8) enables mic access. (Day-one fallback: plain Chrome over HTTPS gets you the mic
    without kiosk lockdown.)

  • Tailscale (or any way to serve trusted HTTPS to the tablet).

  • A Gemini API key (Google AI Studio) for the Live voice.

  • A grounding LLM endpoint that speaks OpenAI-style POST /v1/chat/completions — this can be
    OpenAI, a local runner (Ollama/LM Studio with an OpenAI shim), or a self-hosted agent. It's what
    answers the child's real questions with authority. (Grounding is optional — without it, factual
    questions simply defer to "ask your dad.")

  • Node 22+.

Quick start

# 1. Front-end deps
npm install

# 2. Back-end deps + config. Your Gemini key goes HERE, server-side — the server
#    mints a short-lived ephemeral token per connect, so the key never reaches
#    the browser. Do NOT put it in a root .env; see the note below.
cd server
npm install
cp .env.example .env          # GEMINI_API_KEY, CONSOLE_TOKEN, grounding endpoint, SSL cert paths
cp kid-facts.example.yaml kid-facts.yaml                 # your family facts (gitignored)
cp skills/dad_dossier.example.json skills/dad_dossier.json   # optional dad dossier (gitignored)

# 3. Build the front-end, then run the server (serves build/ over HTTPS)
cd ..
npm run build
cd server && npm start

Don't bake your key into the bundle. A root .env with
REACT_APP_GEMINI_API_KEY still works and is kept only as a dev fallback —
it compiles your key into build/, which is public JavaScript that anyone who
loads the page can read. The server-side path above (GEMINI_API_KEY in
server/.env → /api/gemini_token) is the one to use. See Security model §3.

Then open https://<your-host>:8443 on the tablet, arm kid mode from the console (see SETUP.md),
tap the glyph, and say hello.

➡ Before you run it for real, do SETUP.md — it walks the ~11 things you personalize
(your child's name, your town map for Dad Link, your family facts, your Home Assistant entity, your
safety denylist, etc.) and the tablet/kiosk lockdown steps.

➡ Then do AUTHORING.md — this is the actual project. SETUP gets it running in
forty minutes and leaves you with a very expensive Iron Man soundboard. AUTHORING is the content you
write: the persona, the dad dossier the ladder pays out in, the vocabulary cards, the field missions
and experiments, the trivia, and your family's inside jokes. Everything shipped here is a
[BRACKETED_CAPS] placeholder or a starter set — none of it is anyone's real child.

When something feels wrong: read the transcripts

The voice model talks browser-direct to Google — your server is not in that conversation and
cannot see it. What it keeps is a tee of both sides at server/transcripts/YYYY-MM-DD.jsonl, and
that file is the only real record of what happened. Render it:

python3 scripts/parse_transcript.py server/transcripts/$(date +%F).jsonl --tz -5

Each line carries t0 (when the utterance started) and at (when the server got it). The gap
between them is latency, not silence — which is why "it ignored him" is usually a slow tool
call, and why you'd never find it by reading code. Full method, and the mundane causes to check
after (stale bundle, the charger, the mic's HTTPS context), in SETUP.md §8.

Security model — keeping YOUR child out of YOUR data

Every wall here is built for one job: a child holds this tablet all day, so the child must never
be able to reach anything sensitive — yours or anyone's. When you stand it up, these protections
point at your family's data. Configure them (see SETUP.md); don't weaken them.

  1. The parent is the only operator (fail-closed). Arming/disarming, switching mode, approving a
    memory, overriding a lockout, and reviewing generated trivia all require your CONSOLE_TOKEN.
    The child's same-origin browser has no token, so it cannot do any of these — every gated route
    returns 401 without it. Keep the token secret; it is the master key.
  2. The grounding model must not know your life. ⚠ Most important. A child's live questions go to
    KID_ASK_MODEL. Point it at a memory-less / isolated model — NEVER at a personal assistant
    that has your notes, email, or history, or a child can fish your private life out of it. The
    output passes a family-privacy denylist (KID_ANSWER_DENYLIST — your private names/topics)
    and a child-safety denylist, and anything crossing a line becomes an in-character deferral.
    Those are backstops; the memory-less model is the real wall.
  3. The API key never reaches the browser. The server mints a fresh, short-lived ephemeral
    token     per connect from a GEMINI_API_KEY it holds; the real key isn't in the public client
    bundle. (A static REACT_APP_GEMINI_API_KEY remains only as a dev fallback.)
  4. Child memory is isolated and parent-reviewed. Remembered moments go to a dedicated child
    bank — never a shared/operator bank — and only queue; nothing is kept until you approve it.
  5. Nothing family-specific is in this repo. Family facts (server/kid-facts.yaml), the dad
    dossier, runtime state, transcripts, and all secrets are gitignored — you supply them locally
    from the .example templates. Never commit the real files.

Credits & license

Built on google-gemini/live-api-web-console (Apache-2.0, © Google LLC). This project retains that
license — see LICENSE. The Kid Mode layer (persona, rank ladder, server authority,
Dad Link, safety walls) is the added work described above.