{
  "id": "page:docs-research-realtime-agent-gaps",
  "_kind": "Page",
  "_file": "wiki/docs/research/realtime-agent-gaps.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "sourcePath": "docs/research/realtime-agent-gaps.md",
    "sourceKind": "repo-docs",
    "title": "Realtime agent harness — gap register (voice + video/avatar)",
    "displayName": "Realtime agent harness — gap register (voice + video/avatar)",
    "slug": "docs/research/realtime-agent-gaps",
    "articlePath": "wiki/docs/research/realtime-agent-gaps.md",
    "article": "\n# Realtime agent harness — gap register (voice + video/avatar)\n\n> **Status:** Living gap map for the realtime audio+video avatar agent harness on kradle.\n> **Date:** 2026-06-23.\n> **Companions:** [`realtime-voice-agent-stack.md`](./realtime-voice-agent-stack.md) (architecture), [`voice-governance-bridge-spec.md`](./voice-governance-bridge-spec.md) (bridge + CRD + UX flow), [`g0-rt-live-validation-runbook.md`](./g0-rt-live-validation-runbook.md) (the operator runbook to boot the sidecar into a live room + verify A/V publish + the G13 governance round-trip).\n> **Legend:** **HAS** = real & usable · **PARTIAL** = real but incomplete/unwired · **LACKS** = absent/stub. Seams are `file:line` in `packages/…`.\n\n## 0. The shape of the problem\n\nkradle's **control plane is real and fairly complete**; the **media plane is almost entirely stub**. The harness can already model a meeting agent, dispatch it as a K8s Job with a headless-Chromium Jitsi sidecar, render the agent as a participant tile in the web UI, and expose meeting MCP tools — but the agent currently publishes only Chrome's **fake spinning-ball video + beep** (`--use-fake-device-for-media-stream`), and the agent's reasoning never actually reaches the sidecar at runtime. Closing the harness = (1) fix the **load-bearing runtime gap** (agent↔sidecar socket client), then (2) build the **media plane** (avatar render, real A/V tracks, lipsync, screen-share, tool-driven visuals), then (3) extend the **CRD/UX** surface to declare & drive video capability.\n\n---\n\n## 1. LOAD-BEARING gap (everything else depends on it)\n\n| # | Gap | State | Seam / evidence | Notes |\n|---|---|---|---|---|\n| **G0** | **Agent→sidecar socket client at runtime** | **DONE (impl) · live-pending** | Implemented `jitsi-agent-sidecar/src/ipc-client.js` (`createAgentIpcClient`: connect → NDJSON `command` → `command_result` correlation → `onEvent` stream; no silent fallback) + `bin/agent-ipc-client.mjs` CLI consuming the MCP `{socketPath,command}` descriptor (`kradle/cli/src/mcp-server.js:724-732`). Verified via `runtime.test.js` (9/9, non-socket) + `ipc-client.test.js` (real AF_UNIX socket, runs on Linux CI; platform-skips where AF_UNIX is unbindable). | **Was the load-bearing gap; now built + unit-verified.** Residual = wire it into the in-pod agent wrapper so a dispatched agent actually invokes it (and the live G0-RT meeting boot). |\n| **G0-RT** | **Agent-Job runtime substrate + live meeting boot** | **DEPLOYED green · live join CONVERGING in CI** | Board→agent E2E verified live (memory `project_dispatch_emptydir_agentimage_task`). For the meeting path the prerequisites are now **deployed green to `kradle-staging`**: (a) the **sidecar image** builds+publishes to ACR and is threaded to the controller via `KRADLE_JITSI_AGENT_SIDECAR_IMAGE` (`publish.yml` + `jitsi-agent-bridge.js:16`); (b) **in-cluster Jitsi** runs (`jitsi.install=true`; `kradle-jitsi-subchart-web/-jvb` + `kradle-prosody` live); (c) the **JWT secret-sync fix** — the controller now signs the meeting JWT with the same `KRADLE_JITSI_JWT_SECRET` prosody validates with (it previously fell back to `dev-jitsi-secret` → every dispatched JWT rejected). A repeatable **live E2E** (`scripts/g0-rt-jitsi-e2e.sh` + `.github/workflows/g0-rt-jitsi-e2e.yml` standalone, plus a gated `publish.yml` job via `-f run_g0rt=true`) dispatches a video agent into a meeting and asserts the sidecar joins. | The fast standalone E2E (`gh workflow run g0-rt-jitsi-e2e.yml --ref staging`, ~3 min) drove the live boot through, in order: cluster-auth → jitsi service discovery → namespace → provider/appearance/AgentStack apply (`role: participant`; CRD enum is `observer\\|participant\\|moderator`) → JWT-secret → **test-session auth** (set `KRADLE_TEST_AUTH_SECRET` repo secret) → **permission review** (needs an `AgentServiceAccount` + a `model-provider` `AgentSecretGrant` with `subject:{name}`) → meeting create + status-subresource `Active` patch. **The meeting-dispatch software path is now PROVEN LIVE end-to-end** (closes the long-standing \"no live E2E of the meeting-sidecar dispatch\" gap): dispatch via the kradle **CLI** (`KRADLE_NAMESPACE=kradle-org-<org> node packages/kradle/cli/bin/kradle.mjs dispatch …` — a fresh per-invocation, kubectl-backed controller that reads live k8s, bypassing the unstable BFF's stale snapshot; core is zero-dep so no `npm ci`) → **permission review passes** (needs a real **K8s ServiceAccount** *and* the kradle `AgentServiceAccount` CRD + `model-provider` `AgentSecretGrant{subject:{name}}`) → `getMeeting` reads the `Active` meeting → **agent Job submitted with correct env** (`JITSI_ROOM_URL`→in-cluster web, `JITSI_VIDEO_MODE:publish`, the JWT, the `/tmp` socket) → **pod created**. **Current blocker = CLUSTER CAPACITY, not the harness:** the pod is `Pending`/`FailedScheduling` — `0/4 nodes available: Insufficient cpu` (the control plane + the in-cluster Jitsi stack consume the AKS headroom). **Unblock:** scale the AKS node pool / free CPU; the same E2E (`gh workflow run g0-rt-jitsi-e2e.yml --ref staging`) then schedules the pod. **Likely next blocker once scheduled:** the dispatched `JITSI_JWT` is the **custom `kradle-jitsi.<payload>.<sig>` format** (not a standard `header.payload.sig` JWT) — prosody may reject it at join (the mcp-server token-format issue); then JVB UDP media (X2) is the deeper residual. |\n\n---\n\n## 2. Media plane — sidecar (`packages/kradle/jitsi-agent-sidecar/src`)\n\n| # | Gap | State | Seam | Fix |\n|---|---|---|---|---|\n| G1 | Render avatar in-browser | **DONE (impl) · live-publish pending** | `puppeteer-jitsi-client.connect()` now `injectAvatar()`s when `JITSI_VIDEO_MODE==='publish'`: a `node:http` static server serves `src/browser/{avatar,compositor,publish-effect,lipsync}.js` (promoted byte-identical from `avatar-poc/`), grafted into the live Jitsi page via es-module-shims (`shimMode`) + CDN importmap + `importShim()`, building the avatar from `JITSI_AVATAR_*` env and defining `window.__kradleAvatar`. `verify-sidecar.mjs` S1 confirms a non-blank render headless. Residual: real-GPU TalkingHead quality (cluster). |\n| G2 | Publish generated **video** track | **DONE (impl) · live-publish pending** | `injectAvatar()` builds `createCanvasPublishEffect({canvas,fps})` + `attachToConference` → `localVideoTrack.setEffect(effect)` (the `JitsiStreamPresenterEffect` pattern) off the composited output canvas. `verify-sidecar.mjs` S2 confirms a live video track + `setEffect` on a mock conference. Residual: the real `window.APP.conference` publish into a live Jitsi room (G0-RT). |\n| G3 | Publish **TTS audio** track | **DONE (impl) · live-publish pending** | `audio.js` now routes `speak()` → pluggable `TtsProvider` (mock synthetic default) → audio descriptor; `puppeteer-jitsi-client.publishAudio` builds a Web-Audio `MediaStreamAudioDestinationNode` audio track (avatar-poc `verify-audio.mjs` A1/A2 confirm a live `kind:audio` track headless). Residual: real TTS provider (creds) + the live `window.APP.conference` publish are manual/config. |\n| G4 | **Lipsync** (viseme/mouth sync) | **DONE (impl) · live-publish pending** | `src/browser/lipsync.js` `buildVisemeSchedule` on the `AudioContext` clock + `LipsyncRunner` (X1), now folded into the live sidecar: `publishAudio()` drives lipsync on the shared `window.__kradleAudio.ctx` when the avatar is present and the descriptor carries viseme timing (no silent fallback — a lipsync fault surfaces, never degrades to a plain tone). `verify-sidecar.mjs` S3 confirms the audio track + `replaceTrack`. Residual: real Azure visemes + live publish. |\n| G5 | STT (hear the user) | **PARTIAL (impl) · inbound wiring pending** | `audio.js` `transcribe()` → pluggable `SttProvider` (mock canned default), capability-gated; `runtime.onInboundAudio` routes to the existing `transcript` event. Residual: the page-side inbound-audio tap emits only a signal (no PCM) and isn't wired to `onInboundAudio` yet — real inbound PCM + a real STT provider are manual/live. |\n| G6 | VAD / turn detection | **DONE (impl)** | `audio.js` `detectVoice()` → pluggable `VadProvider` (energy-threshold `local-vad` default); capability-gated; `energy` detail opt-in (keeps the deepEqual success shape). Residual: a real Silero/framework VAD adapter is config-driven. |\n| G7 | Screen-share / VNC into video | **DONE (impl) · live-publish + real-VNC pending** | The compositor now has a real screen layer: `compositor.setScreenSource(el)/clearScreen()` + `drawScreenLayer` composite a source (canvas/video/image) into `#out` (full/inset/region, readiness-guarded), and a real annotation layer (`pushAnnotation`/`clearAnnotations` — text/rect/line/image). `window.__kradleAvatar.startScreenshare({source,url})` feeds it: **noVNC** RFB (lazy CDN-importmap `@novnc/novnc`, ws/wss VNC url) → canvas → screen layer; `getDisplayMedia`→`<video>`; image url→`<img>`. `verify-sidecar.mjs` S6 confirms a synthetic source composites while the avatar base + published track stay live (S5 confirms annotations). The legacy `share_screen` (`window.open`) action is marked **deprecated** in favor of `start_screenshare`. | The real noVNC RFB against a live **websockify/VNC server** (actual desktop pixels) + desktop **input/operate**, `getDisplayMedia` on a real display, and the live `window.APP.conference` publish (G0-RT) stay manual/cluster (X7 throughput). |\n| G8 | New IPC actions for visuals | **DONE (impl) · drives the live avatar** | Added `set_expression/set_posture/play_gesture/look_at/set_view/draw_canvas/start_screenshare/send_video_metadata` to `SUPPORTED_ACTIONS` + `handleCommand` → `jitsi.*` puppeteer methods. With the media plane now wired (G1/G2), `setExpression/setPosture/playGesture/lookAt/setView` mutate the **real** `window.__kradleAvatar` (`verify-sidecar.mjs` S4 confirms `set_expression` changes the rendered mood). Dispatch verified (`runtime.test.js`, exact args). With G7 landed, `draw_canvas`→`compositor.pushAnnotation` (S5) and `start_screenshare`→`compositor.setScreenSource` (S6) now drive the **real** compositor too. | Only `send_video_metadata` remains a no-op — it is the **governed** data-channel path (G13). |\n| G-chat | Inbound `chat` events not emitted | **DONE (impl) · live-pending** | `runtime.js` now emits a normalized `chat` event, and `puppeteer-jitsi-client.connect()` installs the `onEvent` listener it previously **ignored** (the real reason no inbound event flowed). Round-trip verified non-socket. | The in-page Jitsi `message` listener is best-effort/unverified against a live page (real Jitsi API surface) — confirm on a live meeting. |\n\n---\n\n## 3. Control plane — CRDs & controllers (`packages/kradle/core/src`)\n\n| # | Gap | State | Seam | Fix |\n|---|---|---|---|---|\n| G9 | AgentStack **video capability** | **DONE (impl)** | `agent-stack-controller.js` `JitsiCapabilityReady` now validates `capabilities.video:'publish'` + `avatarRef` (resolved against `AgentAppearance`) + video tools + `governedTools⊆tools` + observer-cannot-publish gate (audio/chat path intact). node:test verified. | Residual: `kradle_speak` is allowlisted but not yet MCP-wired. |\n| G10 | Appearance/voice **threaded to sidecar** | **DONE (impl) · live-publish pending** | Closed the resolved-but-undelivered gap: `prepareMeetingContext` now receives `identity` (`agent-dispatch-controller.js:483`), the bridge embeds `meetingContext.{avatar,voice,video}` (bare-stack falls back to `jitsiConfig.avatarRef`, **hard-fail** on unresolvable — no silent fallback), and `createJitsiSidecarContainer` emits `JITSI_VIDEO_MODE`+`JITSI_AVATAR_*`+identity `JITSI_TTS_*`. node:test verified end-to-end. | Residual: the sidecar must actually consume the env to render/publish (G1/G2 live wiring). |\n| G11 | AgentAppearance avatar **model** field | **DONE (impl)** | `resource-model.js` `AgentAppearance` gains optional `renderer/avatarModelUrl/visemeSet/defaultMood/defaultView` (no required-field regression). | — |\n| G12 | JitsiMeeting **media/session status** | **DONE (impl) · live-pop pending** | `jitsi-meeting-controller.js` reconcile adds `status.media/session/transcript/governanceRuns`, **`status.recording` preserved**. node:test verified. | Residual: `governanceRuns` is reserved (`[]`) — live population is the G13 bridge; `media`/`transcript` populate from the live sidecar. |\n| G13 | Governance of consequential visual tools | **DONE (impl) · bridge-delivery + live-pop pending** | The consequential visual tools (`draw_canvas` content, `share_surface`/screen-share start, `send_video_metadata` to external sinks) now route through the babysitter media-governance boundary instead of emitting a direct socket write; cosmetic animation stays fast-path (unchanged). `cli/src/mcp-server.js` `governedToolDescriptor` returns a governance descriptor (`{governed,correlationId,filler,policy,governedProcess,inputs,socketPath}` — **no direct `command`**; hard-deny → `{denied:true}`), gated on the meeting's `governedTools` (G9 validates `governedTools⊆tools`). A pure `PolicyEngine` (`cli/src/policy-engine.js`) decides allow/require-approval/deny with the load-bearing `auth.`/`destroy.` breakpoint prefix. SDK-side (`cli/governance/`, committed, imports the SDK from root node_modules at bridge/verify time only — kradle cli stays zero-dep): a **deterministic** `governed-visual-tool.process.js` (resolveTarget→policyCheck→`auth.`/`destroy.` owner breakpoint→emit approved socket command / denied) + an in-process `run-driver.js`. The socket command is produced **only** by an approved run; the journal is the per-decision audit record. Verified: kradle cli `node --test` 77/77 + SDK-driven `verify-governed-visual.mjs` (approve→command, policy hard-deny→no command, breakpoint-reject→denied, replay determinism). No package/dep/lockfile change. **Bridge delivery (spec §3.2/§3.3) is now built in-place** (`cli/governance/{mcp-http-server,async-delivery}.js`, node:http only): an HTTP MCP server a LiveKit worker connects to with one line (`MCPServerHTTP(url=…)`), where a governed `tools/call` returns immediately with `{correlationId, filler, status:'waiting-approval'}` (fast tools synchronous, unchanged), the G13 run drives out-of-band, and the terminal outcome is delivered via SSE (`GET /mcp/events`) + poll (`GET /governance/:id`); the owner approve/deny channel (`POST /governance/:id/approve|deny`) resolves the run's breakpoint — **the socket command is delivered only after approval** (no default auto-approve, no timeout). Verified `verify-bridge-delivery.mjs` 6/6 over loopback HTTP. | Follow-up: the standalone `@a5c-ai/voice-adapter` package (deferred — owner chose in-place to avoid a lockfile change), a real LiveKit worker connecting + speaking the filler/injecting the SSE outcome, routing owner approval to a real human channel (DTMF/console/Slack), transport hardening (auth/TLS/run GC), threading per-call `context.policy` into the out-of-band re-check (currently defaults; the authoritative deny runs at the MCP boundary with full context, so it fails closed), and **live population** of `JitsiMeeting.status.governanceRuns` from the run journals (G12 residual). |\n\n---\n\n## 4. Surfaces — web UX & MCP\n\n| # | Gap | State | Seam | Fix |\n|---|---|---|---|---|\n| G14 | Stack-builder video/meeting section | **DONE (impl)** | The active `GraphStackBuilder` gains a \"Meeting / Video\" section + provider/avatar/voice/role pickers (new `agents/appearances` + `agents/voices` list routes); pure `buildStackResource` emits `jitsiCapability`/`jitsiConfig`/`jitsiMeetingProviderRef` per modality. A node:test round-trips the emitted spec through the **real** `reconcileStack` G9 validator (True/False). Web suite 353/353. | Residuals (minor): validator doesn't *require* `avatarRef` for video (worth a small follow-up); live React render/POST + the dispatched sidecar publishing are manual. G15 overlay + G17 (genty/ui avatar editor) deferred. |\n| G15 | Agent overlay/controls in call UI | **DONE (impl) · live-render pending** | A pure `deriveAgentOverlay(meeting)` (`web/app/components/jitsi/agent-overlay.js`) reads the real `JitsiMeeting` status (`media.agentTracks`/`session.agents`/`governanceRuns`) to surface which participant is the agent avatar, what it publishes (audio/video/screenshare), and any pending governance approval (a G13 run at phase `waiting-approval`); rendered in `jitsi-meeting-experience.jsx` as an overlay + a minimal mute/hide-agent control (held `api.executeCommand`). Missing status → correct emptiness (no placeholder, no silent fallback). node:test (`jitsi-overlay-spec.test.js`) added to the web suite (357/357). | Live React render of the overlay + the mute/hide control actually muting the iframe tile, and `status.governanceRuns`/`media` populating live, are manual/cluster (ride G13 live-pop + G1/G2/G10 live publish). |\n| G16 | Video MCP tools | **DONE (impl)** | `cli/src/mcp-server.js` adds 9 video tools (`kradle_set_expression`/`play_gesture`/`set_posture`/`look_at`/`set_view`/`draw_canvas`/`publish_video`/`share_surface`/`send_video_metadata`), each returning the `{socketPath,command}` descriptor with role/`requireVideoPublish` gates; `MCP_TOOLS` 33→42. node:test verified. | The descriptor is **written to the socket by the G0 ipc-client** (live wiring), not the tools. |\n| G17 | Agent voice/avatar editor UX | **DONE (impl) · live-render pending** | The existing `genty/ui/.../AgentAppearanceEditor.tsx` is extended with the G11 `AgentAppearance` avatar fields (`renderer`, `avatarModelUrl`, `visemeSet`, `defaultMood`, `defaultView`) + a pure `buildAppearanceSpec`/`parseAppearanceSpec` mapping (omits unset, preserves avatar/emoji/badge/theme); `AgentVoiceEditor` + profile-image editing intact. Targeted vitest 8/8 (round-trip + defaults). | Live React render + the appearance POST whose realtime fields actually drive the sidecar renderer (rides G10 sidecar-env consumption + G1/G2 live publish) are manual/cluster. |\n\n---\n\n## 5. External components to integrate (not in repo — must add)\n\n| # | Component | State | Source / choice |\n|---|---|---|---|\n| G18 | Realtime media server + SIP | **LACKS** | LiveKit server + `livekit/sip` **or** the existing Jitsi (already wired) — Jitsi is the path of least resistance since the sidecar+CRDs target it |\n| G19 | Avatar renderer | **LACKS** | TalkingHead.js (RPM GLB + Three.js) in headless Chromium; Live2D alt (Cubism Core proprietary) |\n| G20 | STT / TTS providers | **LACKS** | Deepgram / faster-whisper (STT); Azure (visemes) / ElevenLabs / Cartesia (TTS) |\n| G21 | noVNC + websockify | **LACKS** | screen-share/desktop-into-video |\n| G22 | (optional) server-side face anim | **LACKS** | NVIDIA Audio2Face-3D (MIT SDK, audio→blendshapes; still needs a renderer) for higher-fidelity lipsync on GPU |\n\n---\n\n## 6. Cross-cutting hard problems / unknowns\n\n| # | Risk | Why it's hard |\n|---|---|---|\n| X1 | **Cross-track A/V sync drift** | Audio + canvas video are separate WebRTC tracks; not auto-lip-synced beyond RTCP. Must originate both from one page/one audio clock; network jitter buffers still add offset. Budget testing. |\n| X2 | **Headless WebGL + WebRTC** | Headless Chromium needs real GPU/ANGLE (SwiftShader is slow); older headless modes broke `mediaDevices`/`captureStream`. Verify under `--headless=new` + GPU on the sidecar host. |\n| X3 | **GPU cost** | One GPU box per concurrent video bot (WebGL composite readback; worse if Audio2Face). Capacity/scheduling concern for the kradle scheduler. |\n| X4 | **Lipsync retargeting accuracy** | Azure ARKit blendshapes are tuned for ARKit, not Oculus/TalkingHead rigs; TalkingHead's rule-based EN mapping ~80%. Retarget quality varies. |\n| X5 | **Governance latency budget** | babysitter governance round-trip (seconds) must hide behind filler speech + async tools; never on the audio/animation hot path. Unbenchmarked — measure `create→iterate→commit` on target disk. |\n| X6 | **lib-jitsi-meet `setEffect`/`replaceTrack` reliability** | Reported multi-second/20s delays, null-old-track failures. Pin a known-good LJM version; test the effect path. |\n| X7 | **noVNC throughput** | Full-desktop RFB over WebSocket at video frame rate is bandwidth-heavy; rate-limit / dirty-region. |\n| X8 | **Licensing** | Live2D Cubism Core proprietary (revenue-gated); RPM sample avatar CC BY-NC; Mixamo raw files no-redistribute; Audio2Face NIM under NVIDIA AI license (not OSS); HeyGen/D-ID closed SaaS. Ship own-licensed assets. |\n\n---\n\n## 7. Recommended build order (dependency-aware)\n\n0. **G0-RT** — confirm the meeting-sidecar dispatch actually boots (the Job runtime already works for board dispatch; build/publish `kradle/jitsi-agent-sidecar:latest` and prove the 2nd container joins a room). Precondition for everything below. **Operator steps + manifests + the landmine checklist are in [`g0-rt-live-validation-runbook.md`](./g0-rt-live-validation-runbook.md).**\n1. **G0** — the agent↔sidecar socket client (unblocks everything; makes today's text chat actually work).\n2. **G3 + G5 + G-chat** — real audio out (TTS) + audio in (STT) + chat events → a working **voice** agent in a meeting (the [voice stack](./realtime-voice-agent-stack.md) MVP).\n3. **G1 + G2 + G4** — avatar render + video-track publish + lipsync → a working **talking-avatar** agent.\n4. **G8 + G16 + G17** — visual tool calls (expression/posture/gesture/canvas) end-to-end (IPC actions + MCP tools + editor).\n5. **G7 + G21** — screen-share / VNC into the video.\n6. **G9 + G10 + G11 + G12 + G14** — CRD video capability + appearance/voice threading + status + stack-builder UX (the full declarative flow).\n7. **G13** — babysitter governance over consequential visual tools (canvas content, screen-share, external metadata).\n8. **X-items** — harden sync/GPU/latency/licensing as each lands.\n\nPrior **voice gaps** (STT/TTS/VAD/WebRTC/SIP from the original voice research) are absorbed here as G3/G5/G6/G18/G20 — they are the same media-plane gaps, now grounded in the kradle sidecar.\n",
    "documents": []
  },
  "outgoingEdges": [],
  "incomingEdges": [
    {
      "from": "page:docs",
      "to": "page:docs-research-realtime-agent-gaps",
      "kind": "contains_page"
    }
  ]
}