{
  "id": "page:docs-research-genty-llm-prompt-caching-plan",
  "_kind": "Page",
  "_file": "wiki/docs/research/genty-llm-prompt-caching-plan.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "sourcePath": "docs/research/genty-llm-prompt-caching-plan.md",
    "sourceKind": "repo-docs",
    "title": "genty-core Vendor-Aware LLM Prompt Caching — Implementation Plan",
    "displayName": "genty-core Vendor-Aware LLM Prompt Caching — Implementation Plan",
    "slug": "docs/research/genty-llm-prompt-caching-plan",
    "articlePath": "wiki/docs/research/genty-llm-prompt-caching-plan.md",
    "article": "\n# genty-core Vendor-Aware LLM Prompt Caching — Implementation Plan\n\nStatus: **plan only** — no source changes. This document specifies an additive design for prompt\ncaching in `packages/genty/core/src/session.ts`. All line numbers below are current as of\n`staging` at commit `743744639` and were verified by reading the file directly.\n\n## 1. Goals & non-goals\n\n### Goals\n\n- Let genty-core opt into vendor-native prompt caching (Anthropic `cache_control`, OpenAI/Azure\n  automatic caching, Google Gemini implicit + explicit caching) on the single completion call\n  path in `callCompletionApi` (`packages/genty/core/src/session.ts:605-703`).\n- Make caching **additive and per-provider opt-in** via `AgentCoreSessionOptions` — a session that\n  does not enable caching must produce byte-identical request bodies to today, and behave\n  identically on every provider.\n- Surface cache-hit/write telemetry (`cache_read`, `cache_creation`, `cached_tokens`, etc.) through\n  the existing `CompletionUsage` / `AgentCorePromptResult.usage` shape so callers can observe cache\n  effectiveness without new APIs.\n- Keep the up-to-50-iteration tool-calling loop (`runCompletionLoop`,\n  `packages/genty/core/src/session.ts:1169-1375`, `MAX_TOOL_LOOP_ITERATIONS = 50` at line 35)\n  untouched in control flow — caching is a request-shaping concern, not a loop-control concern.\n- Follow the repo's \"fallbacks are evil\" rule (`CLAUDE.md`): if a caller enables caching for a\n  provider/config combination that cannot honor it, genty-core must fail loud (throw with a clear\n  message) or explicitly no-op with a logged reason — never silently downgrade to \"it just didn't\n  cache\" without telling the caller why.\n\n### Non-goals\n\n- No change to the tool-calling loop's control flow, convergence-guard thresholds\n  (`MAX_REPEATED_TOOL_CALLS`, `MAX_CONSECUTIVE_TOOL_ERRORS`), or history bookkeeping\n  (`historyEntries`, `AgentCoreHistoryEntry`).\n- No change to `readOpenAiStream` / `readAnthropicStream` SSE event *parsing* logic beyond adding\n  new usage fields — no new event types are introduced.\n- No general-purpose \"cache manager\" abstraction shared across providers. Gemini's explicit-cache\n  resource lifecycle is different enough (create/reference/delete against a REST resource,\n  independent of a single completion call) that this plan scopes it as a separate, later-phase\n  subsystem (see §5.1) rather than forcing it into the same shape as Anthropic/OpenAI/Azure.\n- No change to `resolveEndpoint` provider-detection heuristics\n  (`packages/genty/core/src/session.ts:305-362`) beyond reading new env/config for cache settings.\n- No retrofitting of `packages/babysitter-sdk/src/prompts/strata.ts` — it is cited as *precedent*\n  for the `cache_control` shape, not a dependency to be refactored. Any future unification is out\n  of scope for this plan.\n\n## 2. Config surface\n\n### 2.1 Current shape (baseline, read from `packages/genty/core/src/types.ts`)\n\n`AgentCoreSessionOptions` (`types.ts:83-180`) already carries `model`, `systemPrompt`,\n`appendSystemPrompt`, `customTools`, `backend`, `modelAttestationKey`, and `policyToolGate`. There\nis currently no caching-related field. `CompletionUsage` is defined in `session.ts:111` as\n`type CompletionUsage = NonNullable<AgentCorePromptResult[\"usage\"]>`, and\n`AgentCorePromptResult.usage` (`types.ts:64-70`) is:\n\n```ts\nusage?: {\n  inputTokens: number;\n  outputTokens: number;\n  totalTokens: number;\n  provider?: string;\n  model?: string;\n};\n```\n\nNo cache-specific counters exist today.\n\n### 2.2 Proposed additions to `AgentCoreSessionOptions` (`types.ts`)\n\nAdd one new optional, additive block. Nothing here is required; omitting it must reproduce\ntoday's behavior exactly.\n\n```ts\n/**\n * Opt-in vendor-aware prompt caching. When absent, no caching directives are\n * added to any provider request body (current behavior, byte-identical).\n * Per-provider knobs are independent because each vendor's cache mechanism\n * has a different shape (Anthropic: explicit breakpoints; OpenAI/Azure:\n * automatic, config is advisory only; Gemini: implicit + optional explicit\n * resource).\n */\npromptCaching?: {\n  /** Master switch. Defaults to false. When false, all sub-options are ignored. */\n  enabled: boolean;\n  anthropic?: {\n    /**\n     * Where to place cache_control breakpoints. See §4 for placement\n     * rationale. Defaults to [\"system\", \"tools\"] when enabled.\n     */\n    breakpoints?: Array<\"tools\" | \"system\" | \"history\">;\n    /** cache_control.ttl. Anthropic supports \"5m\" (default) or \"1h\". */\n    ttl?: \"5m\" | \"1h\";\n  };\n  openai?: {\n    /** Forwarded as prompt_cache_key (routing hint only, no-op if unsupported by model). */\n    promptCacheKey?: string;\n  };\n  azure?: {\n    /** Forwarded as prompt_cache_key where the deployment supports it. */\n    promptCacheKey?: string;\n  };\n  gemini?: {\n    /**\n     * \"implicit\" relies on automatic server-side caching (no request\n     * change). \"explicit\" requires an out-of-band CachedContent resource —\n     * see §5.1. Defaults to \"implicit\" when enabled.\n     */\n    mode?: \"implicit\" | \"explicit\";\n    /** Required when mode === \"explicit\"; see §5.1 lifecycle. */\n    cachedContentName?: string;\n    ttl?: string; // e.g. \"3600s\"\n  };\n};\n```\n\nDesign rationale for this shape:\n\n- **One master `enabled` flag** rather than per-provider enable flags, because a given genty-core\n  session already resolves to exactly one provider via `resolveEndpoint`\n  (`session.ts:305-362`) — a caller cannot mix providers mid-session. Per-provider sub-objects\n  hold vendor-specific *tuning*, not independent on/off switches, which avoids the ambiguous case\n  of \"anthropic disabled but openai enabled\" on a session that is actually routed to Anthropic.\n- **Additive to `AgentCoreSessionOptions`**, not a new top-level session constructor parameter,\n  because `AgentCoreSessionHandle` (`session.ts:1077` onward) already threads `this.options`\n  through `runCompletionLoop` → `callCompletionApi` with no other config channel; introducing a\n  second config object would fork the plumbing.\n- **No silent capability fallback.** If `promptCaching.enabled === true` and the resolved\n  `ResolvedEndpoint` (`session.ts:297-303`) is neither `isAnthropic` nor `isAzure`/OpenAI-shaped\n  nor recognized as Gemini (Gemini is not currently a supported `ResolvedEndpoint` branch at all —\n  see §3.4), `callCompletionApi` must throw\n  `Error(\"promptCaching.enabled is set but endpoint <apiBase> has no supported caching path\")`\n  rather than proceed uncached. This is the \"fail loud\" side of the no-fallback rule.\n\n## 3. Per-vendor request-shape changes\n\nAll changes are confined to the three branches inside `callCompletionApi`\n(`session.ts:605-703`): the Anthropic branch (639-656), the Azure branch (657-667), and the plain\nOpenAI-compatible branch (668-679). Gemini has no branch today (see §3.4).\n\n### 3.1 Anthropic (`session.ts:639-656`)\n\nCurrent body construction:\n\n```ts\nconst systemPrompts = request.messages.filter(m => m.role === \"system\").map(m => contentToText(m.content));\nconst nonSystemMsgs = request.messages.filter(m => m.role !== \"system\");\nconst structuredPrompt = buildAnthropicStructuredOutputPrompt(request.structuredOutput);\nconst system = [...systemPrompts, structuredPrompt].filter(Boolean).join(\"\\n\\n\");\nconst baseMessages = nonSystemMsgs.map(m => ({ role: m.role, content: toAnthropicContent(m.content) }));\nconst extra = extraRawMessages.map((m) => ({ role: m.role, content: m.content }));\nbody = JSON.stringify({\n  model: endpoint.model,\n  max_tokens: 16384,\n  stream: true,\n  ...(system ? { system } : {}),\n  ...buildAnthropicTools(request.customTools),\n  messages: [...baseMessages, ...extra],\n});\n```\n\nToday `system` is a single joined **string**. Anthropic's `cache_control` can only be attached to\na **content block**, not a bare string, so caching the system prompt requires switching `system`\nfrom `string` to `Array<{type:\"text\", text:string, cache_control?}>` when caching is enabled\n(Anthropic accepts both shapes for `system`; the array form is additive and does not change\nbehavior for callers who never read the raw body). Insertion points:\n\n1. **New helper** `buildAnthropicSystemBlocks(system: string, cacheEnabled: boolean, ttl)` next to\n   the existing `buildAnthropicStructuredOutputPrompt` (`session.ts:448-466`). When\n   `cacheEnabled` and `\"system\"` is in `breakpoints`, emit\n   `[{ type: \"text\", text: system, cache_control: { type: \"ephemeral\", ...(ttl===\"1h\"?{ttl:\"1h\"}:{}) } }]`;\n   otherwise keep the current bare-string `system` field untouched.\n2. **`buildAnthropicTools`** (`session.ts:592-603`) — add `cache_control` to the **last** tool\n   definition in the array when `\"tools\"` is in `breakpoints`. Anthropic's cache lookback covers\n   everything before and including a breakpoint, so tagging only the final tool entry is\n   sufficient to cache the whole tool block; this mirrors the `strata.ts` pattern of tagging the\n   rendered block, not every sub-part (`strata.ts:305-310`).\n3. **History breakpoint** (optional, `\"history\"` in `breakpoints`) — tag the **last message** in\n   `baseMessages` (i.e., the end of stable prior turns, before `extra`) with `cache_control` when\n   the caller signals the conversation prefix is stable. This is the one insertion point that\n   interacts with the tool loop; see §5.\n\nAnthropic constraint to encode in the helper: **max 4 `cache_control` breakpoints per request**\n(hard vendor limit) — the plan's three breakpoint categories (tools, system, history) plus any\nfuture one must be validated against this cap in the helper, throwing rather than silently\ndropping a breakpoint if a caller somehow requests more than 4.\n\n### 3.2 Azure OpenAI (`session.ts:657-667`)\n\n```ts\nbody = JSON.stringify({\n  model: endpoint.model,\n  messages: [...request.messages.map(toOpenAiMessage), ...extraRawMessages.map(toOpenAiRawMessage)],\n  max_completion_tokens: 16384,\n  stream: true,\n  ...buildOpenAiResponseFormat(request.structuredOutput),\n  ...buildOpenAiTools(request.customTools),\n});\n```\n\nAzure's caching is automatic server-side (on by default, cannot be disabled) — there is **no\nrequest field required** to enable it. The only request-shape change is optionally forwarding\n`prompt_cache_key: options.promptCaching.azure.promptCacheKey` as a spread, e.g.\n`...(cacheKey ? { prompt_cache_key: cacheKey } : {})`, inserted directly in the body object at\nline ~663. Because Azure caching cannot be turned off, `promptCaching.enabled === true` with no\n`azure` sub-config is a valid, meaningful no-op-on-the-wire state — this must NOT throw (contrast\nwith Gemini in §3.4), since the caching is already happening; the config only adds an optional\nrouting hint. Document this asymmetry explicitly in the helper's comment so it isn't mistaken for\nan oversight.\n\n### 3.3 Plain OpenAI-compatible (`session.ts:668-679`)\n\nStructurally identical to the Azure branch (same `toOpenAiMessage`/`buildOpenAiTools` helpers).\nApply the same treatment: optional `prompt_cache_key` from `promptCaching.openai.promptCacheKey`,\nsame \"automatic, cannot disable, cache key is advisory only\" comment. Note this branch is also\nused for non-Azure, non-Anthropic custom endpoints (`agentMuxApiBase` without provider hints,\n`session.ts:333-336`) — the plan makes no assumption that these are truly OpenAI-compatible for\ncaching purposes beyond forwarding the same optional field; if a custom endpoint ignores it, that\nis expected and harmless (it is not a \"supported caching path\" being silently downgraded, it is an\noptional hint being sent to an endpoint that may or may not use it — consistent with OpenAI's own\n\"prompt_cache_key is advisory\" semantics).\n\n### 3.4 Google Gemini — not currently a supported endpoint\n\n`ResolvedEndpoint` (`session.ts:297-303`) and `resolveEndpoint` (`session.ts:305-362`) have no\nGemini branch today: the function only distinguishes `isAzure` / `isAnthropic` / (implicit)\nOpenAI-compatible. This plan does **not** propose adding full Gemini chat-completion support as a\nside effect of a caching plan — that is a separate, larger change (new SSE parser shape, since\nGemini's `generateContent`/`streamGenerateContent` response schema differs from both OpenAI's and\nAnthropic's `readOpenAiStream`/`readAnthropicStream`). Two consequences:\n\n- Per the no-fallback rule, if `promptCaching.gemini` is set on a session whose `resolveEndpoint()`\n  result is not recognized as Gemini, `callCompletionApi` throws — it must never silently ignore\n  Gemini config on a non-Gemini endpoint.\n- The vendor-request-shape work for Gemini in this plan is written as a **forward-looking\n  specification** to land only once/if genty-core gains a Gemini `ResolvedEndpoint` branch and its\n  own stream reader (tracked as a prerequisite, not part of this caching change). Until then,\n  `promptCaching.gemini` remains a documented-but-unimplemented config shape that throws\n  `\"Gemini caching requires Gemini endpoint support, which genty-core does not yet have\"` if set.\n  This keeps the config surface (§2.2) stable for when that prerequisite lands, without pretending\n  the capability exists today.\n\nImplicit-mode request shape (once Gemini support exists): no changes to `generateContent` body —\ncaching is automatic for Gemini 2.5+ when the model is used unmodified; only `usage_metadata`\nparsing changes to surface `cached_content_token_count`.\n\nExplicit-mode request shape: instead of building an inline `body`, the call site must first\nresolve/create a `CachedContent` resource (`POST /v1beta/cachedContents`) out-of-band and then send\n`{ \"cachedContent\": \"cachedContents/{id}\" }` alongside the turn's volatile content. See §5.1 for\nwhy this does not fit the current single-call `callCompletionApi` shape without a new lifecycle\nlayer.\n\n## 4. Cache-breakpoint placement strategy for genty\n\nAligning with the `strata.ts` model (`packages/babysitter-sdk/src/prompts/strata.ts:27`,\n`STRATUM_ORDER = ['stable', 'runtime', 'turnLocal']`) and its `stratumToCacheControl` mapping\n(`strata.ts:284-290`, which tags `stable` and `runtime` as `ephemeral` and leaves `turnLocal`\nuncached), genty-core's completion path has an analogous three-tier structure, even though it does\nnot use `strata.ts` directly (that module lives in `babysitter-sdk`, genty-core is a separate\npackage with its own message-building code):\n\n| Tier | genty-core source | Cache breakpoint? |\n|---|---|---|\n| **stable** | `buildAnthropicTools`/`buildOpenAiTools` output (`session.ts:575-603`) — `request.customTools`, which is fixed per session (`AgentCoreSessionOptions.customTools`, set once at session construction) | Yes — tag last tool entry |\n| **stable/runtime** | `system` block: `options.systemPrompt` + `options.appendSystemPrompt` joined in `buildSystemPrompt` (`session.ts:115-130`), flowing into `request.messages` as a `role: \"system\"` entry consumed at `session.ts:643` | Yes — tag the system block |\n| **runtime** | Prior conversation history fed via `baseMessages` (`session.ts:647`) — the non-system messages already in `request.messages` before this turn's user prompt | Optional — tag the last stable-prefix message only when the caller opts into `\"history\"` breakpoints, since this segment grows every turn and a misplaced breakpoint here wastes writes (see gotcha below) |\n| **turnLocal** | `extraRawMessages` (`session.ts:616`, `648`, `655`) — the tool-call/tool-result turns appended by `runCompletionLoop` for the *current* prompt's tool loop, plus the just-submitted user prompt | No — never cache; this is the volatile tail by construction (fresh every loop iteration) |\n\nRationale mirrors `strata.ts`'s ordering principle (`STRATUM_ORDER`, stable-first) directly:\nAnthropic's own docs specify cache lookback covers the prefix *before* a breakpoint, so\nbreakpoints must be placed after the most-stable, least-frequently-changing content and as early\nin the render order as possible. genty-core's Anthropic body render order is **tools → system →\nmessages** (matches `session.ts:649-656` field order in the `JSON.stringify` call), which already\nlines up with stable-first placement with zero reordering needed — tools and system are\nstructurally first, so tagging their end blocks is sufficient without moving anything.\n\nGotcha specific to genty's tool loop: unlike a single-shot completion, `extraRawMessages` grows on\nevery iteration of `runCompletionLoop` (`session.ts:1231`, `1347`, `1350`) within the same prompt's\n50-iteration budget. A `\"history\"` breakpoint tagging the last `baseMessages` entry stays valid\nacross all iterations of one `runCompletionLoop` call because `baseMessages` itself is fixed for\nthe duration of that call (only `extraRawMessages` mutates) — so per-call cache reuse across tool\niterations is free once implemented. Cross-*prompt* reuse (a fresh `session.prompt()` call reusing\na previous prompt's cached prefix) is a separate question the config's `ttl` addresses (5m default\ncovers back-to-back prompts in the same session; 1h for longer gaps) but is bounded by the strict\nbyte-prefix-match rule — any change to `customTools`, `systemPrompt`, or the history included\ninvalidates the cache regardless of `ttl`.\n\n## 5. Streaming/tool-loop interaction\n\n### 5.1 Do cache markers change `readAnthropicStream`/`readOpenAiStream`?\n\nOnly additively, to parse new usage fields — no change to control flow, SSE event dispatch, or the\nexisting `text`/`toolCalls` extraction.\n\n- **`readAnthropicStream`** (`session.ts:859-981`): the `message_start` handler\n  (`session.ts:900-914`) currently reads `message.usage.input_tokens`. Anthropic's\n  `message_start.message.usage` object also carries `cache_creation_input_tokens` and\n  `cache_read_input_tokens` when caching is active. Extend the usage object built at\n  `session.ts:905-911` (and the `message_delta` handler at `session.ts:931-943`, which currently\n  only reads `output_tokens`) to also read and forward these two fields when present. No new event\n  types, no change to `toolUseBlocks` handling (`session.ts:872-897`, `921-927`).\n- **`readOpenAiStream`** (`session.ts:716-821`): the `chunk.usage` branch\n  (`session.ts:780-790`) currently reads `prompt_tokens`/`completion_tokens`/`total_tokens`.\n  OpenAI/Azure report `usage.prompt_tokens_details.cached_tokens` when caching is active — extend\n  the `chunk` type annotation (`session.ts:742-755`) to include\n  `prompt_tokens_details?: { cached_tokens?: number }` and forward it into the constructed\n  `CompletionUsage`. No change to `toolCallAccumulator` handling (`session.ts:730`, `769-779`).\n\n### 5.2 `CompletionUsage` / `AgentCorePromptResult.usage` extension\n\nAdd optional fields to both `CompletionUsage` (derived from `AgentCorePromptResult.usage`,\n`types.ts:64-70`) and update `mergeUsage` (`session.ts:1001-1014`) to sum them across tool-loop\niterations exactly like `inputTokens`/`outputTokens` are summed today:\n\n```ts\nusage?: {\n  inputTokens: number;\n  outputTokens: number;\n  totalTokens: number;\n  provider?: string;\n  model?: string;\n  cacheReadTokens?: number;      // Anthropic cache_read_input_tokens / OpenAI-Azure cached_tokens\n  cacheWriteTokens?: number;     // Anthropic cache_creation_input_tokens only (OpenAI/Azure don't report writes)\n};\n```\n\n`mergeUsage` gains two more `base.x + next.x` lines guarded the same way `inputTokens` is (treat\nabsent as 0). This is purely additive to an already-optional field, so it does not change the\nshape for callers who never enabled caching.\n\n### 5.3 Does inserting cache markers change the tool-result-feeding logic?\n\nNo, by construction, for Anthropic and OpenAI/Azure. `buildAssistantToolCallMessage`\n(`session.ts:1027-1051`) and `buildToolResultMessages` (`session.ts:1054-1075`) build\n`extraRawMessages` entries, which per §4 are always in the **turnLocal / never-cached** tier — cache\nmarkers are never attached to entries these two functions produce. The only touch point is that\n`callCompletionApi` (called once per loop iteration at `session.ts:1205-1215`) must re-apply the\n*same* stable-tier breakpoints (tools, system, optionally the fixed `baseMessages` prefix) on every\niteration, which it already does naturally since `request` (containing `customTools` and the system\nmessage) is the same `NormalizedCompletionRequest` object across all iterations of one\n`runCompletionLoop` call — no new per-iteration state is needed.\n\n### 5.4 Gemini explicit caching does not fit today's per-call architecture\n\nThis is the one genuinely structural mismatch, and it needs its own subsection because it cannot\nbe solved by editing `callCompletionApi` alone:\n\n- `callCompletionApi` is a stateless, single-request function — it builds a `url`/`headers`/`body`\n  and calls `fetch` exactly once per iteration (`session.ts:681-686`). Anthropic and OpenAI/Azure\n  caching fit this model because the \"cache\" is a passive server-side artifact keyed off request\n  content — no separate resource to create, reference, or delete.\n- Gemini explicit caching is a **stateful external resource**: `POST /v1beta/cachedContents` to\n  create it (returns an id + `expireTime`), then every subsequent `generateContent` call references\n  it via `\"cachedContent\": \"cachedContents/{id}\"`, and callers are billed **storage** for the\n  resource's lifetime regardless of hit rate — so an unmanaged resource is a live cost leak, not\n  just a missed optimization.\n- This resource's lifecycle spans multiple `runCompletionLoop` calls (potentially multiple\n  `session.prompt()` calls sharing one `AgentCoreSessionHandle`), which is a different lifetime\n  than anything `AgentCoreSessionHandle` (`session.ts:1077` onward) currently manages — the handle\n  today has no \"session-scoped external resource that must be cleaned up\" concept at all (compare\n  to `activeAbortController`, which is per-loop-call, not per-session).\n- Recommendation (not designed in full here, flagged as follow-up work gated on Gemini endpoint\n  support existing at all per §3.4): introduce a small resource-lifecycle helper — e.g.\n  `GeminiCacheHandle` created lazily on first use of a session with\n  `promptCaching.gemini.mode === \"explicit\"`, held on `AgentCoreSessionHandle` alongside\n  `activeAbortController`, and explicitly disposed (`DELETE /v1beta/cachedContents/{id}`) in a new\n  `session.dispose()`/`close()` path. No such disposal path exists on `AgentCoreSessionHandle`\n  today — introducing session-level `dispose()` is itself a small breaking-adjacent addition (an\n  optional method, not a required lifecycle change) that should be scoped and reviewed\n  independently of vendor request-shaping. Per the no-fallback rule, if a caller sets\n  `mode: \"explicit\"` without a mechanism to guarantee disposal (e.g. process exit without cleanup),\n  the plan should surface a loud warning at minimum, not silently leak the resource.\n\n## 6. Testing strategy\n\n### 6.1 Unit tests per vendor request-builder\n\nColocate with existing `session.ts` tests (find via the package's `packages/genty/core` test\ndirectory — mirror existing suite naming). For each provider branch:\n\n- **Anthropic**: assert `system` is a bare string when `promptCaching` absent/disabled (regression\n  guard for the byte-identical requirement in §1); assert `system` becomes a one-element array with\n  `cache_control: { type: \"ephemeral\" }` when `breakpoints` includes `\"system\"`; assert the last\n  tool in `buildAnthropicTools` output carries `cache_control` and earlier tools do not, when\n  `breakpoints` includes `\"tools\"`; assert a request with 5 conceptual breakpoints requested throws\n  before hitting the network (validates the 4-breakpoint cap from §3.1); assert `ttl: \"1h\"` is\n  forwarded as `cache_control.ttl` only when configured, defaulting to no `ttl` field (Anthropic's\n  own default) otherwise.\n- **Azure/OpenAI**: assert body has no new fields when `promptCaching` absent; assert\n  `prompt_cache_key` appears only when `promptCaching.azure.promptCacheKey` /\n  `promptCaching.openai.promptCacheKey` is set; assert `promptCaching.enabled: true` with no\n  provider sub-config does **not** throw (the \"automatic, no request change needed\" no-op case from\n  §3.2/§3.3, as opposed to Gemini's throw case).\n- **Gemini (forward-looking)**: assert `promptCaching.gemini` set against a non-Gemini-resolved\n  endpoint throws with the documented message from §3.4, proving the no-fallback guard exists even\n  before full Gemini support lands. This test can be written and should pass today, before any\n  other Gemini work exists.\n- **`mergeUsage`**: unit-test that `cacheReadTokens`/`cacheWriteTokens` sum correctly across two\n  merged `CompletionUsage` objects, including the case where one side is `undefined` (mirrors\n  existing `inputTokens` merge tests).\n- **Stream parsers**: feed a synthetic Anthropic SSE stream whose `message_start` includes\n  `cache_creation_input_tokens`/`cache_read_input_tokens` into `readAnthropicStream` and assert\n  they land in the returned `usage`; same for a synthetic OpenAI-shape chunk with\n  `usage.prompt_tokens_details.cached_tokens` into `readOpenAiStream`.\n\n### 6.2 Verifying cache hits in genty's own usage reporting\n\n- Add a debug-log line (gated behind existing `process.stderr.write(...)` conventions already used\n  in this file, e.g. `session.ts:318`, `345`) inside `runCompletionLoop` after `mergeUsage` when\n  `aggregatedUsage.cacheReadTokens` is present and non-zero, printing a hit-rate-style ratio\n  (`cacheReadTokens / inputTokens`). This gives operators observable confirmation without a new\n  telemetry pipeline.\n- For live/manual verification (per repo convention of live-stack validation over trusting local\n  green — see `MEMORY.md` \"Genty weak-model ceiling\" and \"Live-stack adapters install\" entries):\n  run two back-to-back prompts in the same session with a large, unchanged system prompt/tool set\n  and confirm the second call's `usage.cacheReadTokens` is non-zero against a real Anthropic/OpenAI\n  endpoint — this is the only way to confirm the vendor actually recognized the prefix, since\n  local/unit tests can only prove genty-core sent the right shape, not that the vendor's cache\n  logic accepted it.\n\n## 7. Rollout plan\n\nPhased by implementation risk and external dependency surface, cheapest/most-isolated first:\n\n1. **Phase 1 — Anthropic.** Pure additive request field, no external resource lifecycle, single\n   provider branch (`session.ts:639-656`), immediately testable against the real API. Ship behind\n   `promptCaching.enabled` (default `false`) so existing sessions are unaffected. Add the\n   `cacheReadTokens`/`cacheWriteTokens` usage fields and stream-parser changes from §5.1/§5.2 in\n   the same phase since Anthropic is the only vendor emitting a *write* counter today, and\n   validating it end-to-end needs both sides.\n2. **Phase 2 — OpenAI / Azure.** Automatic, no-request-change-required caching plus the optional\n   `prompt_cache_key` hint (`session.ts:657-679`). Lowest implementation risk of all phases (no\n   conditional body branching beyond one optional spread), but sequenced after Anthropic because\n   the shared usage-field plumbing (§5.2) and its tests will already exist from Phase 1 and this\n   phase only needs to add the `cached_tokens` read path, not invent the merge/report machinery.\n3. **Phase 3 — Gemini implicit caching**, gated entirely on Gemini gaining a `ResolvedEndpoint`\n   branch and its own stream reader (a prerequisite outside this plan's scope, §3.4). Once that\n   prerequisite exists, implicit-mode caching is close to free (no request change, only\n   `usage_metadata.cached_content_token_count` parsing) and should ship alongside/immediately after\n   basic Gemini chat support lands, not as a separate large effort.\n4. **Phase 4 — Gemini explicit caching**, last, because it requires new resource-lifecycle\n   management (§5.4: create/reference/dispose against `POST/PATCH/DELETE /v1beta/cachedContents`)\n   that has no analog in `AgentCoreSessionHandle` today. This phase should be scoped as its own\n   design pass (a `GeminiCacheHandle` + `session.dispose()` addition) rather than an extension of\n   the vendor-request-shaping work in Phases 1-3.\n\nFeature-flag/metrics notes:\n\n- `promptCaching.enabled` (default `false`) is itself the feature flag — no separate env-var gate\n  is needed since the option is per-session and per-caller-controlled already, consistent with how\n  other opt-in behaviors on `AgentCoreSessionOptions` work today (e.g. `customTools`,\n  `modelAttestationKey`).\n- The debug-log hit-rate line from §6.2 is the initial \"metrics\" surface; if usage grows, promoting\n  it to a structured `AgentCoreSessionEvent` (the existing `emit()` mechanism at\n  `session.ts:1377-1380`, used for `text_delta`/`tool_use`/`tool_result` today) is the natural next\n  step — e.g. a `cache_usage` event alongside the existing ones — but is not required for initial\n  rollout since `usage` is already returned synchronously from `prompt()`.\n\n## 8. Open questions / risks\n\n- **Anthropic parallel fan-out cold-start.** If a caller fires multiple `session.prompt()` calls\n  concurrently with an identical stable prefix, all of them can miss the cache (a write is only\n  readable by other requests once the first response begins streaming). `AgentCoreSessionHandle`\n  is single-session/single-loop (`this.activeAbortController` is a single field, `session.ts:1083`)\n  so this only matters if a *caller* runs multiple sessions/handles concurrently with the same\n  system prompt/tools — worth documenting as a known limitation rather than solving in genty-core,\n  since genty-core has no visibility into sibling sessions.\n- **Byte-exact prefix sensitivity.** Any of `customTools`, `systemPrompt`, or `appendSystemPrompt`\n  changing between prompts on the same session invalidates the entire cached prefix (all vendors).\n  Since `appendSystemPrompt` is explicitly documented as \"additional... segments appended before\n  dispatch\" (`types.ts:115-116`) and could vary per-call in caller code, callers who want to\n  benefit from caching need to be told (in the config's JSDoc) to keep these stable across a\n  session's lifetime — this is a documentation/API-contract risk, not a code risk.\n- **Gemini storage billing for explicit caches.** Per the vendor research, explicit caching bills\n  storage per-token-per-hour for the life of the cache regardless of reuse, which can cost *more*\n  than no caching for low-reuse workloads. The `GeminiCacheHandle` design in §5.4 must default to\n  short TTLs and/or require explicit opt-in per session rather than a global default, to avoid\n  surprising operators with idle storage costs — this should be a design constraint carried into\n  Phase 4, not solved here.\n- **`prompt_cache_key` cardinality (OpenAI/Azure).** OpenAI's docs note caching scope is\n  per-organization; a `promptCacheKey` set to something highly unique per call (e.g. a random UUID\n  per prompt) would effectively partition the cache away from reuse. The config JSDoc should warn\n  callers to set a stable key (e.g. per-session, per-agent-role) rather than per-call.\n- **Whether `\"history\"` breakpoints (§3.1 item 3, §4) are worth the complexity.** Unlike system/\n  tools, the conversation history grows every turn, so a naive \"tag the last message\" strategy\n  needs care about *which* message is tagged (the last one **before** the current turn's volatile\n  tail) to avoid constantly moving the breakpoint and forcing cache-write churn instead of reuse.\n  This plan recommends shipping tools+system breakpoints first (Phase 1) and treating history\n  breakpoints as an experimental follow-up gated on real usage data showing history reuse is\n  actually happening across prompts (not just within one tool loop, where it's free per §4's\n  gotcha analysis).\n- **No current Gemini support at all.** This plan's Gemini sections (§3.4, §5.4, Phases 3-4) are\n  necessarily speculative about the exact `ResolvedEndpoint`/stream-reader shape, since that\n  prerequisite doesn't exist. They should be revisited once/if a genty-core Gemini integration\n  design exists, rather than treated as final.\n",
    "documents": []
  },
  "outgoingEdges": [],
  "incomingEdges": [
    {
      "from": "page:docs",
      "to": "page:docs-research-genty-llm-prompt-caching-plan",
      "kind": "contains_page"
    }
  ]
}