{
  "id": "page:docs-development-07-live-stack-qa-guide",
  "_kind": "Page",
  "_file": "wiki/docs/development/07-live-stack-qa-guide.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "sourcePath": "docs/development/07-live-stack-qa-guide.md",
    "sourceKind": "repo-docs",
    "title": "07 — Live Stack QA Guide",
    "displayName": "07 — Live Stack QA Guide",
    "slug": "docs/development/07-live-stack-qa-guide",
    "articlePath": "wiki/docs/development/07-live-stack-qa-guide.md",
    "article": "\n# 07 — Live Stack QA Guide\n\n## Overview\n\nThe live-stack workflow validates end-to-end harness compatibility by running real agent/model combinations through the transport-adapter proxy. Each scenario installs a harness adapter, configures a model provider, and executes a babysitter process to confirm that the full pipeline — from agent CLI through transport-adapter translation to model API and back — works correctly.\n\nThese tests catch integration regressions that unit tests and mocked pipelines cannot: mismatched streaming formats, incorrect provider translations, broken harness adapters, and transport-adapter proxy routing failures.\n\n## Push Defaults\n\nEvery push to `staging` or `main` automatically runs the following matrix:\n\n| Agent | Model | Modes | Install | Process Mode |\n|-------|-------|-------|---------|--------------|\n| Claude Code | foundry-gpt55 | NI + bridged-interactive | vanilla | — |\n| Codex | google-gemini31 | NI + bridged-interactive | vanilla | — |\n| Pi | foundry-Kimi-K2.6 | NI + bridged-interactive | vanilla | — |\n| Claude Code | foundry-gpt55 | interactive | bp | predefined |\n| Codex | google-gemini31 | interactive | bp | predefined |\n| Claude Code | foundry-gpt55 | interactive | bp | resume |\n| Codex | google-gemini31 | interactive | bp | resume |\n| Claude Code | foundry-gpt55 | bridged-hooks | bp | predefined |\n| Codex | google-gemini31 | bridged-hooks | bp | predefined |\n\nThe first three rows test harness adapters against different model providers. The BP rows verify the babysitter-plugin integration across predefined and resume process modes.\n\n## Dispatch Inputs\n\nThe workflow supports three inputs for `workflow_dispatch`:\n\n| Input | Description | Default |\n|-------|-------------|---------|\n| `ref` | Branch or ref to check out and test | dispatched ref (staging) |\n| `os` | Runner OS: `ubuntu-latest-l`, `macos-latest`, `windows-latest` | `ubuntu-latest-l` |\n| `matrix` | JSON array of test combinations | push defaults |\n\nThe `ref` input allows testing a PR branch's code without merging. Both the build and test jobs check out the specified ref.\n\nThe `os` input allows cross-platform testing on macOS or Windows runners. When running interactive mode on macOS CI (no TTY available), the test runner automatically falls back to `--bridge-interactive` mode.\n\n## When to Run Manual Dispatch\n\nTrigger a manual dispatch in the following situations:\n\n- **After adding a new harness adapter** — run the new agent against at least one model/mode combination to confirm it integrates correctly.\n- **After changing transport-adapter proxy logic** — run a broad sweep to verify no existing translations broke.\n- **After modifying provider translations** — target the affected provider across multiple agents.\n- **Before releases (full matrix)** — run all agents against all providers to establish a complete compatibility baseline.\n- **When investigating a specific harness failure** — isolate the failing combination for faster iteration.\n- **Cross-platform validation** — dispatch with `os=macos-latest` to test macOS compatibility.\n\n## How to Dispatch\n\n### Via GitHub UI\n\n1. Go to **Actions** in the repository.\n2. Select the **Live Stack** workflow.\n3. Click **Run workflow**.\n4. Fill in the inputs: `ref`, `os`, and `matrix` JSON.\n\n### Via CLI\n\n```bash\ngh workflow run live-stack.yml --ref staging \\\n  -f os=ubuntu-latest-l \\\n  -f ref=feat/my-branch \\\n  -f 'matrix=[{\"agent\":\"codex\",\"model\":\"foundry-gpt55\",\"mode\":\"interactive\",\"install\":\"bp\",\"live\":true,\"process_mode\":\"predefined\"}]'\n```\n\n### Matrix JSON Format\n\nEach entry in the array defines one scenario:\n\n```json\n[\n  {\n    \"agent\": \"<name>\",\n    \"model\": \"<model-id>\",\n    \"mode\": \"<mode>\",\n    \"install\": \"<vanilla|bp>\",\n    \"live\": true,\n    \"process_mode\": \"<predefined|create|resume>\"\n  }\n]\n```\n\n| Field | Required | Description |\n|-------|----------|-------------|\n| `agent` | yes | Harness adapter name (see Available Axes below) |\n| `model` | yes | Model provider and identifier |\n| `mode` | yes | Interaction mode |\n| `install` | yes | `vanilla` for adapters only, `bp` for babysitter-plugin |\n| `live` | yes | `true` for real model calls, `false` for mock/dry-run |\n| `process_mode` | bp only | `predefined`, `create`, or `resume` |\n\n## Available Axes\n\n### Agents\n\n`claude`, `codex`, `pi`, `gemini`, `copilot`, `hermes`, `cursor`, `opencode`, `openclaw`, `omp`, `droid`, `amp`\n\n### Models\n\n`foundry-gpt55`, `foundry-gpt54mini`, `google-gemini31`, `anthropic-sonnet46`, `foundry-deepseek`\n\n### Modes\n\n| Mode | Description |\n|------|-------------|\n| `ni` | Non-interactive — agent runs to completion with no user input |\n| `bridged-interactive` | Interactive prompts bridged through transport-adapter |\n| `interactive` | Native interactive mode (BP install only) |\n| `bridged-hooks` | Hook events bridged through transport-adapter (BP install only) |\n\n### Install\n\n| Value | Description |\n|-------|-------------|\n| `vanilla` | Installs adapters only; tests raw harness adapter compatibility |\n| `bp` | Installs babysitter-plugin; tests full plugin integration |\n\n### Process Mode (BP only)\n\n| Value | Description |\n|-------|-------------|\n| `predefined` | Uses the existing `summarize-translate-test.mjs` process fixture |\n| `create` | Agent creates a new process definition during the run |\n| `resume` | Resumes a stalled run from a pre-populated journal fixture (15 events: outline + summaries done, first translation pending) |\n\n### Runner OS\n\n| Value | Description |\n|-------|-------------|\n| `ubuntu-latest-l` | Default Linux runner (large) |\n| `macos-latest` | macOS ARM64 runner — interactive mode auto-bridges when no TTY |\n| `windows-latest` | Windows runner |\n\n### Live\n\n| Value | Description |\n|-------|-------------|\n| `true` | Makes real model API calls through the provider |\n| `false` | Mock/dry-run mode for testing pipeline mechanics without API costs |\n\n## Common Dispatch Examples\n\n### Test a single harness\n\n```json\n[{\"agent\":\"hermes\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true}]\n```\n\n### Test a PR branch\n\n```bash\ngh workflow run live-stack.yml --ref staging -f ref=feat/my-branch \\\n  -f 'matrix=[{\"agent\":\"codex\",\"model\":\"foundry-gpt55\",\"mode\":\"interactive\",\"install\":\"bp\",\"live\":true}]'\n```\n\n### macOS compatibility check\n\n```bash\ngh workflow run live-stack.yml --ref staging -f os=macos-latest \\\n  -f 'matrix=[{\"agent\":\"codex\",\"model\":\"foundry-gpt55\",\"mode\":\"interactive\",\"install\":\"bp\",\"live\":true}]'\n```\n\n### BP resume mode\n\nTest babysitter resume from a stalled run:\n\n```json\n[{\"agent\":\"claude\",\"model\":\"foundry-gpt55\",\"mode\":\"interactive\",\"install\":\"bp\",\"live\":true,\"process_mode\":\"resume\"}]\n```\n\n### BP create mode\n\nTest babysitter-plugin with on-the-fly process creation:\n\n```json\n[{\"agent\":\"claude\",\"model\":\"foundry-gpt55\",\"mode\":\"interactive\",\"install\":\"bp\",\"live\":true,\"process_mode\":\"create\"}]\n```\n\n### Full harness sweep\n\nRun all agents against foundry to validate every adapter:\n\n```json\n[\n  {\"agent\":\"claude\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true},\n  {\"agent\":\"codex\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true},\n  {\"agent\":\"pi\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true},\n  {\"agent\":\"gemini\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true},\n  {\"agent\":\"copilot\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true},\n  {\"agent\":\"hermes\",\"model\":\"foundry-gpt55\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true}\n]\n```\n\n### Anthropic direct\n\nTest Claude Code against the Anthropic provider directly (bypasses foundry):\n\n```json\n[{\"agent\":\"claude\",\"model\":\"anthropic-sonnet46\",\"mode\":\"ni\",\"install\":\"vanilla\",\"live\":true}]\n```\n\n## QA Evidence Wiki\n\nLive-stack test results are tracked on the **[QA Evidence wiki page](https://github.com/a5c-ai/babysitter/wiki/QA-Evidence)**. This page maintains a comprehensive matrix of all agent/model/OS/mode combinations with links to passing CI jobs.\n\n### How to update the wiki\n\n1. After a successful live-stack run, find the passing job ID in the GitHub Actions UI or via CLI:\n   ```bash\n   gh api repos/a5c-ai/babysitter/actions/runs/<RUN_ID>/jobs \\\n     --jq '.jobs[] | select(.conclusion == \"success\") | \"\\(.id)\\t\\(.name)\"'\n   ```\n2. Clone the wiki repo: `git clone https://github.com/a5c-ai/babysitter.wiki.git`\n3. Edit `QA-Evidence.md` — replace `—` with `[PASS](https://github.com/a5c-ai/babysitter/actions/runs/<RUN_ID>/job/<JOB_ID>)` in the appropriate cell.\n4. Commit and push: `git add QA-Evidence.md && git commit -m \"Add <description>\" && git push`\n\n### Wiki structure\n\nThe wiki organizes evidence by:\n- **Test type**: Vanilla NI, Vanilla BI, BP/Predefined Interactive, BP/Predefined Bridged-Hooks, BP/Create Interactive, BP/Create Bridged-Hooks, BP/Resume Interactive, BP/Resume Bridged-Hooks\n- **Model**: gpt-5.5, gpt-5.4-mini, claude-sonnet-4-6, gemini-3.5-flash, Kimi-K2.6\n- **Agent**: claude-code, codex, pi, gemini-cli, hermes, cursor-cli, copilot-cli, opencode\n- **OS**: Ubuntu, macOS, Windows\n\nEach cell links to a specific CI job that demonstrated a successful pass. The wiki also tracks key fixes applied during the stabilization effort.\n\n## Reading Results\n\nThe **Live Stack Report** job runs after all scenarios complete. It generates a summary table with the following columns:\n\n| Column | Description |\n|--------|-------------|\n| Agent | Harness adapter used |\n| Provider | Model provider |\n| Model | Model identifier |\n| Mode | Interaction mode |\n| Process Mode | predefined, create, or resume |\n| Runtime | Execution duration |\n| Status | Pass/fail result |\n\nFailed scenarios include expandable details with error logs, transport-adapter traces, and the last agent output before failure. Look for these when triaging:\n\n- **Transport errors** — usually indicate a proxy routing or translation issue.\n- **Timeout failures** — may indicate a hung agent or unresponsive model endpoint.\n- **Assertion failures** — the agent completed but produced unexpected output.\n- **posix_spawnp / tcgetattr errors** — PTY allocation failures, typically on macOS CI runners.\n\n## Concurrency\n\n- **Push runs** share a branch-based concurrency group with `cancel-in-progress` enabled. A new push to the same branch cancels any in-flight push run.\n- **Dispatch runs** each receive a unique concurrency group. They are never cancelled by push runs or other dispatch runs.\n\nThis means you can safely dispatch a manual run while a push-triggered run is in progress — neither will interfere with the other.\n\n## Known Limitations\n\n- **Cursor requires `CURSOR_API_KEY`** — this secret is not provisioned in the default CI environment. Cursor scenarios will fail unless the key is added to the repository secrets.\n- **Pi NI requires `--mode json` flag** — the Pi harness adapter must pass `--mode json` for non-interactive runs. This is handled automatically by the adapter, but be aware of it when debugging Pi NI failures.\n- **Some harnesses install via pip/curl, not npm** — `hermes` and `cursor` (among others) are installed through pip or curl rather than npm. Their installation steps take longer and depend on external package registries outside the npm ecosystem.\n- **macOS CI runners lack TTY devices** — neither `node-pty` nor the macOS `script` command can allocate a PTY on GitHub Actions macOS runners. Interactive mode tests auto-fall back to `--bridge-interactive` on macOS.\n",
    "documents": []
  },
  "outgoingEdges": [],
  "incomingEdges": [
    {
      "from": "page:docs-development",
      "to": "page:docs-development-07-live-stack-qa-guide",
      "kind": "contains_page"
    }
  ]
}