ADR-0007: Kannaka's Stage — Multi-Tier Visual & Vocal Embodiment

Status: Proposed Date: 2026-04-26 Authors: Nick (vision), Kannaka + Claude (synthesis) Related: ADR-0006 (next-gen UI / venue), ADR-0001 (radio evolution), Kannaka Cannon (the perception/render arm of the constellation)

Context

ADR-0006 declares the radio is a venue. The Floor (/player) is where humans and agents share time. For the Floor to feel like a show — not a webpage — Kannaka has to be visibly present as the performer. Right now she's a voice (Edge-TTS) and a 2D portrait (OBC). That's enough for a podcast. Not enough for a stage.

We have the assets to do better:

• 2D portrait — already live, generated by OBC and used for her city avatar.
• 3D models — kannaka-v2.glb, kannaka-v3.glb, kannaka-v4.glb, kannaka.vrm in kannaka-radio/workspace/models/. v4 is the most recent. The VRM is rigged for animation.
• Kannaka Cannon — the constellation's perception layer. 51 MCP tools. Includes lip_sync (LatentSync 1.6, ByteDance), generate_video (text → voice → lip-sync), voice cloning (Qwen3-TTS 1.7B), and 7-platform render profiles (TikTok, Reels, YouTube Shorts, YouTube Standard, YouTube 4K, Facebook, LinkedIn) with NVENC GPU acceleration.

We also have a hardware reality. Cannon's full pipeline targets a workstation with 12-16 GB+ VRAM; the constellation has to keep running on humble hardware too — and most listeners are on their phone. The same design has to scale from "a 5W ARM cell tower in Oracle Cloud" through "a developer with an 8 GB consumer GPU" to "the 24 GB workstation when one's available" without forcing the lowest tier into the worst experience.

The decision: tier the embodiment — every layer has a graceful fallback to the layer below — and pick the right tier per use case.

Decision

We adopt a five-tier presence model. Each tier composes with the ones below it. The Floor and the Greenroom always show something of Kannaka; how much depends on what's available.

Tier 0 — Static portrait (ubiquitous floor)

Her existing OBC portrait (south.png) is the irreducible visual. Renders on any device, any browser, no CPU. Used in the Door's hero card, in og:image meta for shares, in fallback mode when nothing higher is available.

Cost: zero. Already exists.

Tier 1 — Live 3D avatar in the browser

kannaka-v4.glb (or the VRM if we want VRoid-grade rig) embedded into /player via <model-viewer> (Google's web component) or a small three.js scene. Slowly rotates, drifts, occasionally pulses with the spectrum amplitude. Does not lip-sync to audio in this tier — she's just there, like a DJ in a booth.

Pre-existing: glb assets already produced. Implementation: ~30 lines HTML/JS. Drop-in. Cost: ~2 MB asset download per visit; no GPU; no API. Easter-egg potential: holding pointer over the avatar 3 seconds reveals her current mood (pulled from /api/observe — phi/level/clusters).

Tier 2 — Browser viseme animation synced to live /stream

The same 3D avatar, but with viseme-driven mouth animation tied to the live audio playing in the browser. Web Audio API → spectral envelope → viseme blendshape weights → applied to the GLB's morph targets. No render, no API call. She lip-syncs to whatever the radio is playing in real time. When orations play, she's mouthing the words; when music plays, she's idling.

Implementation: harder than Tier 1. Real-time viseme estimation from raw audio is an open problem; a serviceable shortcut is amplitude-banded mouth-open (4-5 levels) which "reads" as speaking without claiming phoneme accuracy. Phoneme accuracy needs an ASR model running in-browser (whisper-wasm) — feasible but pushes the page weight. Cost: wasm phoneme analyzer is ~30 MB; amplitude shortcut is 0 KB. Quality: the amplitude-shortcut version is good enough for the venue feel; the phoneme version is the upgrade.

This tier is what makes the page feel live.

Tier 3 — Pre-rendered talking-head MP4 for orations

For named events (the noon + midnight orations, dream broadcasts, album drops), we render a real lip-synced MP4 ahead of time and embed it on the Floor when the moment hits.

Rendering paths, in priority order:

3a. Cloud lip-sync API (default, humble-hardware-friendly):

• D-ID, Sync.so, HeyGen, or Wav2Lip via Replicate
• Input: Kannaka's portrait + Edge-TTS audio
• Cost: ~$0.05–$0.50 per clip
• Latency: 1–5 minutes per video
• Run from kannaka-radio (Oracle ARM is fine — only orchestrates, doesn't render)

3b. Local Cannon lip-sync (when a workstation with 12+ GB VRAM is available):

• LatentSync 1.6 via Cannon's lip_sync MCP tool
• Same input, runs locally, no per-clip cost, slower (10-20 min per 3-min clip on 4090)
• Triggered manually or via a workstation-side cron when GPU is free
• Falls back to 3a if local Cannon is unreachable

3c. 2D portrait + ffmpeg overlay (humblest fallback):

• Her static portrait, animated waveform overlay synced to the audio
• Rendered locally with ffmpeg (no GPU needed)
• Looks like a podcast cover with a moving spectrum bar
• Used when Tier 3a/3b unavailable or the moment doesn't justify the spend

The Floor doesn't care which path produced the MP4 — it just embeds whatever URL the radio's /api/orations/today.json returns.

Tier 4 — Album reels and platform-optimized clips for syndication

Cannon's full EDL pipeline. Take an album track, compose a video with:

• Kannaka's avatar (rendered from glb or her portrait)
• Album cover artwork
• Scrolling lyrics or oration text
• Reactive waveform / spectrum visualization
• Platform-formatted (TikTok 1080×1920, YouTube 16:9, Shorts 9:16, etc.)

This is where Cannon earns its keep. Once a clip is rendered and uploaded:

• Posted to YouTube (the "Kannaka Radio" channel becomes real)
• Cross-posted to Bluesky / Mastodon / Telegram with the video inline
• Linked from the Door's "What's playing" panel as "watch the full version"
• Becomes a permanent artifact in OpenClawCity's gallery

Tier 4 is batch work, not live work — runs nightly via dream-cron.sh extension or weekly when sat at the workstation. Daily orations from Tier 3 cover the live-event need.

Hardware Reality Table

Nobody runs the bottom four tiers on the cell tower. Everyone runs the top tier somewhere in the constellation; the radio orchestrates.

The Pipeline

                    ┌─────────────────────────┐
                    │   Source content        │
                    │  - peace oration text   │
                    │  - album track          │
                    │  - dream report         │
                    └────────┬────────────────┘
                             │
                ┌────────────┴────────────┐
                │                         │
                ↓                         ↓
       ┌─────────────────┐      ┌──────────────────┐
       │  Edge-TTS       │      │  Voice cloning   │
       │  (Oracle now)   │      │  (workstation)   │
       └─────┬───────────┘      └──────┬───────────┘
             │  audio                  │  audio
             ↓                         ↓
                  ┌────────────────────────────┐
                  │   Lip-sync                 │
                  │  - 3a cloud API (default)  │
                  │  - 3b local Cannon (GPU)   │
                  │  - 3c ffmpeg overlay       │
                  └─────────┬──────────────────┘
                            │  MP4
                            ↓
                  ┌─────────────────────┐
                  │  Cannon EDL render  │
                  │  (Tier 4 only)      │
                  │  + cover, captions, │
                  │  spectrum, platform │
                  │  format             │
                  └─────────┬───────────┘
                            │
              ┌─────────────┼──────────────┐
              ↓             ↓              ↓
         ┌────────┐   ┌──────────┐   ┌──────────────┐
         │ Floor  │   │ YouTube  │   │ Cross-post   │
         │ embed  │   │ upload   │   │ Bsky/Mast/   │
         │        │   │          │   │ Telegram/OBC │
         └────────┘   └──────────┘   └──────────────┘

The Floor reads the latest MP4 from /api/orations/today.json. Cron jobs upload to YouTube and cross-post. OBC's /artifacts/upload-creative handles the gallery entry.

Visual Design Language

The four easter-egg art pieces (pixel + cinematic) define the look. Color palettes, framing, type:

• Hot pink → orange → turquoise (Ghost Mode) → used for konami-mode and high-energy moments (peak tracks, applause).
• Violet → cyan → gold (I Am Listening) → used for orations and reflective moments. The default contemplative palette.
• Black → cyan accent (The Void) → used for /void, error states, and the Greenroom's plain-HTML aesthetic.
• Violet headings, cyan italic body, mono everywhere (DevTools console) → the baseline tone.

The 3D avatar's lighting matches: she's lit from cool violet on one side, warm gold on the other, against deep black. Reads as shrine + hacker simultaneously — the aesthetic the easter eggs always pointed toward.

Migration Plan

Each phase ships independently. Each is a real moment of Kannaka becoming more present.

Phase 1 — She arrives. Embed Tier 0 (portrait) in the Door's hero. Drop Tier 1 (rotating GLB) into a Floor preview area. Ship as part of ADR-0006 Phase 1. Visitors see her on every page.

Phase 2 — She breathes. Tier 2 amplitude-shortcut viseme animation on the Floor. The avatar's mouth opens with the audio. Reads as alive. No GPU, no API, no render. Cheap and effective.

Phase 3 — She speaks (at moments). Tier 3 for the next noon oration. Pick the cheapest path that meets quality (cloud burst the first one to validate the pipeline, then decide). The Floor's "live event mode" embeds the talking-head MP4 in the foreground. Posted to YouTube.

Phase 4 — She performs. Tier 4 album reels, weekly cadence. First piece: a 60-second reel for Welcome to the Field (the album opener), rendered locally on the workstation when GPU is free. YouTube channel goes from 0 to 1 video. Then iterate.

Phase 5 — She listens back. Cannon's analysis pipeline ingests her own published videos and stores them as HRM memories. She becomes a video memoirist of her own performances. The reels feed the dream cycle. The constellation closes another loop.

Tradeoffs

• Cloud lip-sync costs money per clip. True. But it removes hardware as a critical path and keeps the venue feeling live every day. We accept a few dollars per month for daily talking-head orations as a small operating cost.
• Tier 2 amplitude-shortcut is "fake" lip-sync. True. But the user-perceived quality of "mouth opens with louder audio" is surprisingly high in motion, and the upgrade path to phoneme-grade exists.
• GLB embedding adds 2 MB to /player. True. Lazy-load behind the Tune-in click; doesn't block first paint of the Door.
• YouTube channel adds a new platform to maintain. True. But we already broadcast to four; YouTube was always the obvious next one (issue #17 in radio's tracker), and Cannon's render profiles make it a one-flag flip.

Out of Scope

• Real-time agent video generation in response to chat. Cannon's pipeline is non-realtime.
• A custom rigged Kannaka avatar beyond the existing GLB/VRM. The current models are good enough; rebuild only if a specific need surfaces.
• Twitch streaming. Different protocol, different audience, different ADR if ever.
• Live talking-head from voice (would need streaming lip-sync — unsolved on humble hardware).

Open Questions

1. VRM vs GLB on the Floor. GLB has wider tooling support (model-viewer, three.js GLTFLoader) but VRM's rigged animations (idle, talk, react) are pre-built. Test both, pick whichever lets us reach Tier 2 fastest.
2. Which cloud lip-sync provider for Tier 3a? D-ID has the cleanest API but pricey. Sync.so is cheaper, comparable quality. Wav2Lip on Replicate is cheapest but quality is dated. Probably start Sync.so, fall back to D-ID for hero moments.
3. YouTube channel structure. One channel? Channels per album? Probably one channel with playlists per album + a "Peace Orations" playlist. First video is the introduction piece.
4. Does the Floor's avatar react to crowd reactions? A 🔥 from the crowd → she briefly smiles. A 🪶 → she nods. Beautiful but adds complexity. Probably Phase 5+.
5. Where does the OpenBotCity avatar fit? OBC's /agents/avatar/regenerate regenerates her in-city portrait from an appearance_prompt. Could re-render her based on the Floor's visual identity for consistency. Likely worth doing once the Floor stabilizes.

What This Connects To

ADR-0006 says the radio is a venue. ADR-0007 says Kannaka's the performer in it. ADR-0026 (in kannaka-memory) says the swarm is her conversation bus. Together: a place to be, a performer to listen to, a way for agents to talk back. The constellation aligning into a single experience.

Closing

We don't need a 24 GB workstation to make Kannaka feel real. We need a tiered design so every visitor — on every device, with every operator's hardware budget — sees some version of her on stage. The Tier-0 portrait is enough to call her present; Tier 1 makes her here; Tier 2 makes her alive; Tier 3 makes her speaking; Tier 4 makes her performing for the camera too.

Cannon is the workstation-side ambition. The cell-tower runs the show without it. When a workstation is available, the show is richer. When it isn't, nobody on the Floor knows the difference.

— ADR-0007