I.
Wiki article
Reading · 10 mindocs/testing/stack-permutations
Stack Permutations reference
The test strategy must treat the stack as modular. A valid E2E does not need every layer, and some layer combinations are invalid even if the names sound related.
Continue reading
Nearby pages in the same section.
Stack Permutations
The test strategy must treat the stack as modular. A valid E2E does not need every layer, and some layer combinations are invalid even if the names sound related.
Layer Map
| Layer | Package or surface | Owns | Does not own |
|---|---|---|---|
| Core Babysitter SDK | packages/babysitter-sdk, babysitter run:*, task:*, hook:*, plugin:* | Event-sourced runs, task effects, process state, generic plugin registry, SDK harness install commands | Model session UI, adapters adapter registry, provider transport implementation |
| SDK harness setup | babysitter harness:install, babysitter harness:install-plugin | Installing external harness CLIs where supported and installing Babysitter harness plugins | agent-platform runtime behavior |
| Babysitter-agent runtime | packages/genty/platform runtime CLI | Runtime orchestration UX, model-backed planning/execution, genty-core path, adapters bridge for external harness invocation | Harness plugin installation and setup commands |
| Adapters core | packages/adapters/core, @a5c-ai/adapters | Adapter registry, createClient().run, sessions, workspaces, plugin manager, runtime hooks, provider/model config | Babysitter run journal ownership |
| Adapters adapters | packages/adapters/adapters | Per-agent spawn/programmatic adapters, capabilities, session parsing, adapter plugin APIs when supported | Generic Babysitter process orchestration |
| Transport-adapter | packages/transport-adapter | Harness-facing provider protocol routes, local proxy runtime lifecycle, proxy auth, runtime env injection, passthrough forwarding, streaming/non-streaming response shape, cancellation, timeout, and metrics/cache visibility | Installing harnesses/plugins, normalizing hooks, owning Babysitter journals, or proving adapters adapter/session semantics without a consumer |
| Hooks-adapter | packages/adapters/hooks/* | Normalizing raw hook payloads and merge/policy behavior across harnesses | Adapters runtime hook dispatch and SDK stop-hook iteration policy |
| Agent-core | packages/genty/core | Programmatic model session backend and tool-call loop used by internal/runtime paths | External harness plugin installation |
| Agent-plugins-adapter | packages/extensions-adapter | Plugin target discovery and plugin target contracts | Runtime session execution |
Primary E2E Paths
| Path | Entry point | Required setup | What it proves | What it must not claim |
|---|---|---|---|---|
| SDK run-loop E2E | babysitter run:create, run:iterate, task:post | Fixture process and optional mocked hooks | Process state, effects, journal replay, stop-hook continuation | Provider or external harness behavior |
| SDK harness/plugin setup E2E | babysitter harness:install, babysitter harness:install-plugin | Temporary workspace and installer fixtures or real installer runner | Harness install delegation, plugin installer package behavior, idempotent manifests | agent-platform runtime correctness |
| Adapters adapter/session E2E | adapters run <agent> or createClient().run({ agent }) | Adapter fixtures or real agent CLI and credentials | Adapter events, session lifecycle, model/provider config, runtime hooks | Babysitter process journal correctness unless a plugin invokes Babysitter |
| Adapters plugin E2E | adapters plugin ... or client.plugins.* where adapter supports plugins | Adapter with supportsPlugins, plugin manifest/marketplace fixture or real plugin target | Agent-native plugin install/list/uninstall and plugin event behavior | Universal plugin support across all agents |
| Babysitter plugin through adapters E2E | Adapters starts an external harness session after the Babysitter harness plugin is installed | Harness-specific Babysitter plugin installed by SDK installer or native plugin path, then adapters run <agent> | The plugin command such as /babysitter:call creates a Babysitter run, completes it, and hook/stop behavior is visible from the harness session | agent-platform install/setup behavior |
| Babysitter-agent runtime E2E | agent-platform runtime commands | Preinstalled or mocked model backend; no setup command inside the test | Runtime planning/orchestration, selected backend, run lifecycle, task posting, genty-core or adapters bridge behavior | Harness plugin installation |
| Transport-adapter E2E | adapters-proxy, startTransportMuxRuntime, applyTransportMuxToHarnessEnv, or adapters launch --with-proxy* | Local route fixture, genty-core stream, or adapters external-harness launch that needs a proxy bridge | Route/codec contract, proxy auth, env injection, launch proxy decision, streaming/non-streaming response shape, cancellation, timeout, passthrough, metrics/cache artifacts | Plugin install, harness install, hook normalization, or Babysitter run lifecycle by itself |
| Hooks-adapter E2E | Hook adapter CLI/core normalizer | Raw hook payload fixtures or redacted live payloads | Hook normalization, merge policy, fail-open/fail-closed behavior | Adapters session lifecycle by itself |
Transport-Adapter Valid Permutations
Transport-adapter is the carrier/proxy seam between a harness-facing protocol and a target provider/runtime. It can be tested alone with local fixtures, or as a bridge started by adapters launch, but it is not a plugin manager, harness installer, hook adapter, or Babysitter run owner.
| Permutation | Lane | Entry point | Required assertions |
|---|---|---|---|
| Package route/codec fixture | No-model | createTransportMuxApp or adapters-proxy with fixture engine | /health, /v1/models, /metrics, /cache/stats, /v1/count_tokens, /v1/messages, /v1/chat/completions, /v1/responses, /v1beta/models/*, /v1/projects/*, /converse, /models/chat/completions, and /passthrough/* return the expected protocol shapes and errors |
| Runtime env bridge | No-model | startTransportMuxRuntime and applyTransportMuxToHarnessEnv | ADAPTERS_PROXY_BASE_URL, ADAPTERS_PROXY_AUTH_TOKEN, and provider-specific base URL/API key variables are injected only for the exposed transport, with token values redacted in artifacts |
| Adapters launch decision | No-model | resolveLaunchPlan and launch dry-run fixtures | Native provider, proxy forced, proxy if-needed, and proxy forbidden cases produce the expected proxyNeeded, proxyReason, and exposed transport |
| genty-core stream through transport | No-model and model-backed | Agent-core event stream consumed through transport-adapter | Fixture or live deltas, final event, cancellation, timeout, and usage metadata survive transport framing |
| External harness through adapters proxy | Model-backed | adapters launch <harness> <provider> --with-proxy or --with-proxy-if-needed | Real Codex/Claude-compatible harness traffic uses the local proxy URL, emits a redacted launch plan, and completes a sentinel stream |
| Passthrough provider bridge | No-model first, model-backed only when justified | /passthrough/* with configured apiBase | Path/query/body forwarding, auth propagation, upstream failure mapping, and timeout behavior are visible without leaking provider secrets |
Capability-Gated Adapter Matrix
| Agent or harness | Adapters adapter mapping | Current plugin-manager expectation | Runtime-hook expectation | Valid live permutations |
|---|---|---|---|---|
claude-code / claude | claude-code maps to claude | Valid where the Claude adapter exposes plugin APIs | Native/runtime hook coverage including stop hook is valid | Adapters session, adapters plugin manager, Babysitter plugin through adapters, agent-platform external-harness bridge |
codex | codex maps to codex | Capability-gated; current Codex adapter reports supportsPlugins: false, so do not require adapters client.plugins.* for Codex | Runtime hook fixtures are valid; live plugin manager install is not assumed | Adapters session, SDK harness plugin installer, Babysitter plugin through Codex only after installer/native plugin support is proven, agent-platform external-harness bridge |
gemini-cli / gemini | gemini-cli maps to gemini | Capability-gated by adapter | Runtime hook fixture first, live after adapter support is proven | Adapters session and SDK installer smoke; plugin E2E only after capability proof |
genty-core | Not an adapters external harness mapping | No harness plugin install | Programmatic event hooks through owning layer only | Babysitter-agent internal/programmatic runtime, transport-adapter with genty-core stream |
pi | Intentionally not adapters in agent-platform mapping | SDK plugin installer may exist, but runtime path is dire../core-like | Do not route through adapters bridge | Direct S../platform path only |
babysitter adapter in adapters | Adapters can target Babysitter as an adapter | Babysitter plugin manager is generic SDK plugin registry, not external harness plugin install | Adapter parses Babysitter event output | Adapters consuming Babysitter output; separate from agent-platform runtime setup |
Invalid Combinations
| Invalid combination | Why it is invalid |
|---|---|
Babysitter-agent E2E that starts with babysitter harness:install or harness:install-plugin | That tests SDK harness setup, not agent-platform runtime behavior |
Adapters plugin-manager test that requires Codex plugin install without checking supportsPlugins | Current Codex adapter reports plugin manager support as false |
| Transport-adapter test that asserts plugin installation | Transport-adapter carries harness-facing provider traffic; SDK harness setup or adapters plugin APIs own plugin installation |
Transport-adapter test that runs babysitter harness:install | Harness install belongs to SDK harness setup, not the proxy runtime |
| Transport-adapter test that asserts hook normalization | Hooks-adapter owns hook payload normalization; transport-adapter may only carry traffic adjacent to a hook-emitting harness |
| Transport-adapter test that claims Babysitter run completion by itself | Babysitter SDK or agent-platform owns run creation, task posting, and terminal journal state |
| Hooks-adapter fixture that claims full adapters session coverage | Hooks-adapter normalizes hook payloads; adapters owns session lifecycle |
| genty-core path routed through adapters external-harness mapping | The agent-platform map explicitly excludes genty-core and pi from adapters external harness mapping |
/babysitter:call plugin smoke that only checks final assistant text | It must assert Babysitter run ID, run events, terminal state, and hook evidence |
Minimum Permutation Set
The rebuilt strategy should implement these before claiming broad E2E coverage:
| ID | Lane | Stack | Required evidence |
|---|---|---|---|
| P1 | No-model | SDK run loop + mocked stop hook | run:create, pending task, task:post, run:iterate, completed proof, hook log |
| P2 | No-model | SDK harness installer + plugin installer dry-runs | JSON install plan, plugin target, idempotency fixture |
| P3 | No-model | Adapters core + mock adapter + runtime hooks | session_start, prompt/input, session_end, stop-hook decision fixture |
| P4 | No-model | Adapters PluginManager + plugin-capable adapter fixture | list/install/uninstall/update behavior and capability errors for non-plugin agents |
| P5 | No-model | Transport-adapter route/codec fixture | supported route matrix, auth failure, invalid JSON, count_tokens supported/unsupported, streaming and non-streaming response artifacts |
| P5a | No-model | Transport-adapter runtime env bridge + adapters launch decision | redacted env diff, proxy config, proxyNeeded/proxyReason, forced/if-needed/native/forbidden cases |
| P6 | No-model | Hooks-adapter raw payload fixtures | normalized stop/session/tool events and merge-policy artifact |
| P7 | Model-backed | Babysitter-agent + genty-core backend | created run, planned task, posted result, terminal state, redacted model trace |
| P8 | Model-backed | Babysitter-agent + external harness bridge | agent-platform call/invoke, adapters mapped session events, terminal result, no install steps |
| P9 | Model-backed | Adapters + Claude + Babysitter plugin | harness/plugin precondition evidence, adapters run claude, /babysitter:call, Babysitter run completion, stop-hook evidence |
| P10 | Model-backed/capability-gated | Adapters + Codex + Babysitter plugin | Only enabled after plugin install support is proven; otherwise assert skip reason from capability gate |
| P11 | Model-backed | Transport-adapter + genty-core stream | live or credential-gated genty-core deltas carried over transport-adapter, cancellation/timeout behavior, redacted provider metadata |
| P12 | Model-backed | Adapters external harness + transport-adapter proxy | adapters launch starts transport-adapter, harness uses proxy env, sentinel stream completes, metrics snapshot and redacted launch plan are uploaded |
| P13 | No-model | Adapters hooks + hooks-adapter bridge for claude-code, codex, pi | adapters hooks add/handle, a5c-hooks-adapter invoke, normalized phase evidence, no Babysitter SDK calls, no provider credentials |
| P14 | No-model | Pipeline-owned stack matrix across adapters-mocks and real-agent CLI shims for claude, codex, pi, and gemini | adapters install --dry-run, profile-backed launch/run, transport-adapter mock-model request evidence, and optional hooks-adapter normalized phase artifact from no_model_mock_matrix |
Each implementation slice should name which permutation IDs it covers. If a job covers only setup, it should not be labeled as runtime E2E.
Agent-Adapter Live Install Modes
The live external-harness matrix has two valid adapters paths:
| Mode | Valid targets | Installer responsibility | Prompt responsibility | Lifecycle responsibility |
|---|---|---|---|---|
babysitter-plugin | claude-code via claude, codex, gemini-cli via gemini, pi | adapters install <target> installs or verifies the harness CLI; the local Babysitter SDK and generated Babysitter plugin package are installed before launch | The launch prompt is a Babysitter command, for example /babysitter:call ... | Must prove Babysitter run creation, effects, journals/task artifacts, native stop hook execution, hooks-adapter normalization, adapters session, transport trace, and provider trace |
vanilla | claude, codex, gemini, pi, babysitter | adapters install <target> only | The launch prompt is a normal non-Babysitter sentinel prompt | Must prove adapters session/launch, transport trace, and provider trace; it must not claim plugin-driven external-harness hook coverage; agent-platform rows may additionally assert genty-core-backed Babysitter runtime evidence when required |
These are different integration paths. babysitter-plugin validates plugin-mediated Babysitter lifecycle behavior through an external harness; vanilla validates the same adapters install/launch/provider path without Babysitter plugin setup.