{
  "id": "page:library-mlops",
  "_kind": "Page",
  "_file": "wiki/library/mlops.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "title": "MLOps (Library)",
    "displayName": "MLOps (Library)",
    "slug": "library/mlops",
    "articlePath": "wiki/library/mlops.md",
    "article": "\n# MLOps\n\nFlagship model-lifecycle for the library: dataset governance intake (parallel per-dataset lineage/consent/retention checks) -> eval-harness design -> executed training/eval runs -> an adversarial eval-review gate that RE-RUNS a sampled eval and diffs metrics -> a policy-gated model promotion with an executed serving smoke -> drift-monitoring setup with an executed drift-detection stub -> an adversarial drift-review gate -> a drift path with severity-routed escalation and a policy-gated rollback/retirement -> kip-backed model-registry memory. This is a brand-new specialization directory (verified: no prior `mlops` dir anywhere in `library/`).\n\n## Composition map — callable upstream training stages (NOT superseded)\n\n`model-lifecycle.js` consumes a **trained candidate** (`modelVersion` + `artifactRef`) and owns the governance / eval / promotion / drift / retirement lifecycle **around** it. Training itself is a pre-bar point task that can be delegated to either data-science-ml near-miss:\n\n| Upstream stage | Role |\n|---|---|\n| [`data-science-ml/model-training-pipeline.js`](../data-science-ml/model-training-pipeline.js) | Hyperparameter tuning + experiment tracking producing the candidate model artifact that model-lifecycle P3 evaluates and promotes |\n| [`data-science-ml/automl-pipeline.js`](../data-science-ml/automl-pipeline.js) | Alternate: automated algorithm selection / ensembling producing a candidate; feeds the same P3 eval-harness inlet |\n\nThese are mapped as callable upstream stages — **NOT superseded, NOT re-implemented**, nothing deprecated. `mlo.training-run` (P3, optional, gated on `retrain`) is the delegation seam.\n\n## Module table — `model-lifecycle.js` exports\n\n| Export | Kind | Purpose |\n|---|---|---|\n| `process(inputs, ctx)` | orchestrator | The flagship lifecycle, phases P0–P8 |\n| `MODEL_STAGES` | frozen const | `['development','staging','production']` — ordered lifecycle stages |\n| `STAGE_PROMOTION_POLICY` | frozen const | Per-target-stage entry gate + accountable expert (lookup via `stagePromotionPolicy`) |\n| `DATASET_GOVERNANCE_CHECKS` | frozen const | `['lineage','consent','retention']` — the three per-dataset checks (dsar-lifecycle shape) |\n| `DRIFT_SEVERITIES` | frozen const | `['SEV1','SEV2','SEV3','SEV4']` — mirrored from release-lifecycle severity routing |\n| `DRIFT_ROUTING` | frozen const | Drift escalation routing per severity (lookup via `driftRouting`) |\n| `stagePromotionPolicy(stage)` | helper | Promotion-policy lookup — **throws** on unknown stage (no fallback policy) |\n| `driftRouting(severity, request?)` | helper | Routing lookup — **throws** on unknown severity and on `escalationExpert` requests for immediate-rollback severities (no fallback route) |\n| `assertDriftSeverity(value, source)` | helper | Accepts SEV1..SEV4 or `'none'`; anything else **throws** naming the source |\n| `governanceCheckLabel(check)` | helper | Validates a check name against `DATASET_GOVERNANCE_CHECKS`; **throws** on unknown check |\n| `datasetGovernanceCheckTask` | agent task | `mlo.dataset-governance-check` — runs lineage/consent/retention for one dataset (fanned out per dataset) |\n| `retentionExecutionTask` | agent task | `mlo.retention-execution` — executes the approved retention action; only inside `dataset-retention-action` approved |\n| `evalHarnessDesignTask` | agent task | `mlo.eval-harness-design` — authors the benchmark harness + regression-threshold table |\n| `trainingRunTask` | agent task | `mlo.training-run` — optionally (re)trains the candidate; delegable to the data-science-ml near-misses |\n| `evalRunTask` | agent task | `mlo.eval-run` — actually runs one benchmark suite; one instance per evalSuite via `ctx.parallel` |\n| `promotionDeployTask` | agent task | `mlo.promotion-deploy` — promotes exactly the evaluated version; only inside `model-promotion-approval` approved |\n| `promotionVerificationTask` | agent task | `mlo.promotion-verification` — executes a serving smoke proving the promoted version serves |\n| `driftMonitorSetupTask` | agent task | `mlo.drift-monitor-setup` — configures detectors + hooks, captures baseline, runs the drift-detection stub |\n| `driftTriageTask` | agent task | `mlo.drift-triage` — SEV1..SEV4 classification grounded in the executed drift metrics |\n| `rollbackExecutionTask` | agent task | `mlo.rollback-execution` — restores `currentProductionRef` / retires the superseded version; only inside `model-rollback-approval` approved |\n| `rollbackVerificationTask` | agent task | `mlo.rollback-verification` — executed probes proving the restored version serves and the drift symptom is gone |\n\n## Style note\n\nAll tasks are Style-A `kind: 'agent'` (zero `kind: 'shell'`), with per-effect `io` paths (`tasks/<effectId>/input.json|result.json`) and `labels`, and every gate / verification / executed-run output schema declares `evidence { type: 'array', minItems: 1 }`. Gate combinators (`routedBreakpoint`, `adversarialGate`, `kipRecall`, `kipAssert`) are imported from [`../common-utilities/routed-gate-combinators.js`](../common-utilities/routed-gate-combinators.js), not redefined. Timeline, `breakpointsHit`, and `autoApprovals` are accumulated in the orchestrator only — agents never write the timeline.\n\n## Stage model\n\n### Lifecycle stages (`MODEL_STAGES`, verbatim)\n\n`['development', 'staging', 'production']` — promotion advances one stage toward production; `promotionTargetStage` must be a member beyond `development` (default `production`). An unknown value throws (no fallback stage).\n\n### Promotion table (`STAGE_PROMOTION_POLICY`, verbatim)\n\n| Target stage | Entry gate | Expert |\n|---|---|---|\n| `staging` | `model-promotion-approval` | ml-engineering-lead |\n| `production` | `model-promotion-approval` | ml-engineering-lead |\n\nLookups go through `stagePromotionPolicy(stage)`, which **throws** on an unknown stage — there is no fallback promotion policy.\n\n### Drift routing table (`DRIFT_ROUTING`, verbatim)\n\n| Severity | Escalation path | Escalation expert |\n|---|---|---|\n| SEV1 | `immediate-rollback` | — (straight to the `model-rollback-approval` gate; expert lookup throws) |\n| SEV2 | `immediate-rollback` | — (straight to the `model-rollback-approval` gate; expert lookup throws) |\n| SEV3 | `remediation-choice` | ml-engineering-lead |\n| SEV4 | `remediation-choice` | ml-engineering-lead |\n\nAll lookups (`stagePromotionPolicy`, `driftRouting`, `assertDriftSeverity`, `governanceCheckLabel`) **throw** naming the source on any unknown enum — there are no fallback rows anywhere.\n\n## Policy-gated actions\n\nThree actions are policy-gated. Convention: **`breakpointId` = actionId**, strategy `single`.\n\n| actionId | Expert | Tags | Raised when | Rejection behavior |\n|---|---|---|---|---|\n| `dataset-retention-action` | data-governance-officer | `['policy-gated','mlops','dataset-governance']` | P1, only when a dataset governance check flags a deletion/retention-enforcement action; payload carries the dataset, proposedAction, check details, and priorKnowledge governance facts | `retentionExecutionTask` never invoked; the action is recorded not-executed; if the un-actioned dataset is a required training/eval set the lifecycle **fails closed before eval** |\n| `model-promotion-approval` | ml-engineering-lead | `['policy-gated','mlops','promotion']` | P5, only after a passed `mlo.eval-review` gate; payload carries eval metrics, per-threshold pass/fail, eval-review evidence, governance clearance, target stage | Run ends `success:false`, nothing promoted; `promotionDeployTask` never invoked (no alternate path) |\n| `model-rollback-approval` | ml-engineering-lead | `['policy-gated','mlops','<sev>' | 'retirement']` | P7 drift path (SEV1/SEV2 immediately; SEV3/SEV4 after the remediation-choice picks rollback), and for retirement of a superseded production version | Run ends `success:false`, model left at current version, state surfaced; `rollbackExecutionTask` never invoked |\n\n**Fail-closed posture:** there is no alternate execution path around a gate — the retention, promotion, and rollback executors are invoked **only** inside `gate.approved === true` branches, each with an explicit code comment that no other call site exists. **No gate in this process sets `autoApproveAfterN`**, and all three policy gates carry explicit code comments stating it must never be added. Any harness-level auto-approval is surfaced in `outputs.autoApprovals` (`{ breakpointId, phase, at }`), which is **always present** in outputs, possibly empty.\n\n## Quality gates\n\n### `mlo.eval-review` (P4) — RE-RUNS a sampled eval, diffs metrics\n\nRuns over the eval report (per-suite metrics + deterministic threshold pass/fail), with the harness and evalSuites reachable in context. Failure (including an owner-rejected escalation) ends the run **before** `model-promotion-approval` is ever raised (fail closed).\n\n| Critic | Focus |\n|---|---|\n| `sampled-eval-reexecution-critic` | RE-RUNS a sampled subset of an evalSuite itself and DIFFS the fresh metrics against the reported metrics — raw re-execution outputs cited per sampled metric; a citation of the reported number without a fresh run is NOT evidence |\n| `regression-threshold-critic` | Recomputes every metric against `regressionThresholds` (min/max/baseline/maxRegression) and flags any breach the run under-reported — the recomputation is cited |\n| `eval-integrity-critic` | The eval datasets are the governed holdout with no train/eval leakage — cross-checked against the P1 lineage clearance; datasets checked are cited |\n\n### `mlo.drift-review` (P6) — RE-RUNS the drift-detection stub, fires the hooks\n\nRuns over the drift-monitor spec + executed drift-detection run. A live breach in the executed run routes into the P7 drift path.\n\n| Critic | Focus |\n|---|---|\n| `drift-detection-reexecution-critic` | RE-RUNS the drift-detection stub itself over baselineWindow vs checkWindow and DIFFS its drift scores against the reported ones — raw executed outputs cited; a read of the reported drift score is not evidence |\n| `alerting-hook-critic` | Proves each configured alerting hook ACTUALLY FIRES on a synthetic drift breach (executed) — the fired-alert output cited per hook |\n\n**IRON-LAW (executed evidence only):** the eval-review gate must re-run a sampled eval and diff; the drift-review gate must re-run the drift-detection stub and fire the hooks — reading a report or spec is not evidence, and `passed:true` with empty evidence is rejected by the combinator. Fix budget: `maxFixAttempts` (default 2) rounds of the built-in `gateFixerTask`; on exhaustion the combinator escalates to the owner via a routed breakpoint (`mlo.eval-review.gate-escalation` / `mlo.drift-review.gate-escalation`). A model **never promotes on a read-only review**.\n\n## Drift path\n\nEntered only when the executed drift-detection run surfaces a live breach. `driftTriageTask` classifies SEV1..SEV4 grounded in the executed drift metrics (recommendation only). Then `DRIFT_ROUTING`:\n\n- **SEV1 / SEV2** -> straight to the `model-rollback-approval` gate.\n- **SEV3 / SEV4** -> one non-policy `mlo.drift.remediation-choice` breakpoint first (accept-drift/roll-forward vs rollback, expert ml-engineering-lead) — the **only non-policy breakpoint** in this process (sparse-breakpoint rule: the call is genuinely ambiguous at low severity). A `roll-forward` response ends the run `success:false` (drift accepted, no rollback gate raised); any other response proceeds to the gate.\n\nOn `model-rollback-approval` approved, `rollbackExecutionTask` restores exactly `currentProductionRef` and `rollbackVerificationTask` executes serving probes proving the restored version serves and the drift symptom is gone (bounded by `driftPolicy.maxRollbackAttempts`, default 1). **Retirement** of a superseded prior production version is the same gate surface (`model-rollback-approval`, tagged `retirement`): a clean promotion to `production` that supersedes a non-null `currentProductionRef` requests retirement of the old version through this gate.\n\n## kip model-registry memory\n\n- **Recall (P0)**: `kipRecall(ctx, { kipDir, topic: 'model registry: <modelName>@<modelVersion>', kipModel, kind: 'mlops-model-registry' })` — prior model performance, promotion decisions, and drift incidents threaded as `priorKnowledge` into every downstream agent. An empty store is initialized and reported as `factCount: 0`, never an error.\n- **Assert at close (P8)**, facts built deterministically in the orchestrator (never in an agent), subject `model:<modelName>@<modelVersion>`:\n  - `{ predicate: 'has-version', object: <modelVersion> }`\n  - `{ predicate: 'outcome', object: 'promoted'|'failed'|'rolled-back', props: { evalReviewPassed, promoted, driftDetected } }`\n  - one per eval suite: `{ predicate: 'eval-metric', object: <suite>, props: { thresholdsPassed } }`\n  - `{ predicate: 'promotion-decision', object: 'promoted-to-<stage>'|'not-promoted', props: { approved, verified } }`\n  - only when the drift path ran: `{ predicate: 'drift-incident', object: <triage.summary>, props: { severity, rollbackVerified } }` and `{ predicate: 'rollback-lesson', object: '<severity>: <rationale>' }`\n\nBoth touchpoints are wrapped in `if (kipEnabled)` (default `true`). The assert facts are unconditionally non-empty when reached — the has-version and outcome facts always exist.\n\n## Inputs / outputs reference\n\nMirrors the JSDoc `@inputs` / `@outputs` in `model-lifecycle.js`. Required: `model { modelName, modelVersion, artifactRef }`, a non-empty `datasets[]` (each `{ name, uri, purpose }`, purpose `train`|`eval`|`holdout`), a non-empty `evalSuites[]` (each `{ name, dataset, metrics[] }`), and a `regressionThresholds` object mapping every referenced metric to a threshold (a referenced metric with no threshold **throws** before any run). Optional: `promotionTargetStage` (default `production`), `retrain` (default `false`), `driftPolicy { maxRollbackAttempts, baselineWindow, checkWindow }`, `maxFixAttempts` (default 2), `kipEnabled`/`kipDir`/`kipModel`, `artifactsDir`.\n\n## Usage\n\n```bash\nbabysitter run:create \\\n  --process-file library/specializations/mlops/model-lifecycle.js \\\n  --inputs '{\n    \"model\": {\n      \"modelName\": \"fraud-scorer\",\n      \"modelVersion\": \"2.4.0\",\n      \"artifactRef\": \"s3://models/fraud-scorer/2.4.0/model.pt\",\n      \"currentProductionRef\": \"s3://models/fraud-scorer/2.3.1/model.pt\"\n    },\n    \"datasets\": [\n      { \"name\": \"txn-train\", \"uri\": \"s3://data/txn/train\", \"purpose\": \"train\", \"lineageRef\": \"dvc://txn@train\" },\n      { \"name\": \"txn-holdout\", \"uri\": \"s3://data/txn/holdout\", \"purpose\": \"holdout\", \"consentBasis\": \"contract\" }\n    ],\n    \"evalSuites\": [\n      { \"name\": \"accuracy-suite\", \"dataset\": \"txn-holdout\", \"metrics\": [\"auc\", \"precision\"] }\n    ],\n    \"regressionThresholds\": {\n      \"auc\": { \"min\": 0.9, \"baseline\": 0.94, \"maxRegression\": 0.01 },\n      \"precision\": { \"min\": 0.85 }\n    },\n    \"promotionTargetStage\": \"production\"\n  }'\n```\n\nFor this run: the two datasets clear lineage/consent/retention in parallel (a flagged retention action would gate `dataset-retention-action` with the data-governance-officer), the eval harness is authored and every metric is checked for a threshold, `accuracy-suite` runs against the candidate and is scored deterministically, the `mlo.eval-review` gate re-runs a sampled eval and diffs it, `model-promotion-approval` (ml-engineering-lead) gates promotion to `production`, an executed serving smoke verifies it, `mlo.drift-review` re-runs the drift-detection stub, and — because a prior `currentProductionRef` exists — retirement of `2.3.1` is requested through the `model-rollback-approval` gate tagged `retirement`.\n\n## Non-interactive runs\n\nNothing policy-gated auto-approves **by design** — no gate in this process sets `autoApproveAfterN`, and the three policy gates must never gain it. If a non-interactive harness auto-approves a breakpoint at its own level, that approval is recorded in `outputs.autoApprovals` as `{ breakpointId, phase, at }` with its phase provenance, so the fail-closed posture stays auditable. `autoApprovals` is always present in outputs, even when empty.\n",
    "documents": [
      "specialization:mlops"
    ]
  },
  "outgoingEdges": [
    {
      "from": "page:library-mlops",
      "to": "specialization:mlops",
      "kind": "documents"
    }
  ],
  "incomingEdges": [
    {
      "from": "page:index",
      "to": "page:library-mlops",
      "kind": "contains_page"
    }
  ]
}