Agentic AI Atlas

II.

LibraryProcess overview

lib-process:data-engineering-analytics--data-product-lifecycle-workflow

Reference · live

specializations/data-engineering-analytics/data-product-lifecycle-workflow overview

Flagship data-product lifecycle: requirement + data-contract definition -> parallel source discovery/profiling (a source that CANNOT be profiled FAILS the run — no assumed schema) + lineage mapping -> parallel model/pipeline build across the composed point tasks -> parallel EXECUTED data-quality and contract test families -> two adversarial gates whose critics RE-EXECUTE the quality suite and RE-COMPUTE profile stats and row/metric reconciliation against the source of truth (a green dbt run is NOT evidence) -> policy-gated destructive backfill and PII-field exposure -> policy-gated contract-breaking-change (raised ONLY when the contract diff classifies as breaking) -> policy-gated production cutover with executed post-cutover reconciliation -> catalog/contract publish -> SLA + freshness monitoring with an executed probe and severity-routed breach handling -> policy-gated IRREVERSIBLE dataset deprecation -> kip-backed data-product memory. Ownership boundary: this process owns datasets, pipelines and published data contracts; ../mlops/model-lifecycle.js owns trained-model governance, eval, promotion, drift and retirement. A gold-tier product published here is a legitimate `datasets` entry for model-lifecycle; neither supersedes the other.

LibraryProcessOutgoing · 18Incoming · 0

Attributes

displayName

specializations/data-engineering-analytics/data-product-lifecycle-workflow

description

libraryPath

library/specializations/data-engineering-analytics/data-product-lifecycle-workflow.js

specialization

data-engineering-analytics

example

await orchestrate('specializations/data-engineering-analytics/data-product-lifecycle-workflow', { dataProduct: { name: 'orders-gold', version: '2.0.0', tier: 'gold', owner: 'analytics-platform', warehouse: 'snowflake', publishedContractRef: 'contracts/orders-gold@1.4.0.md', }, sources: [ { name: 'raw_orders', uri: 'raw.public.orders', system: 'postgres-cdc', expectedGrain: 'one row per order' }, { name: 'raw_customers', uri: 'raw.public.customers', system: 'postgres-cdc' }, ], models: [ { name: 'dim_customer', pattern: 'scd', grain: 'one row per customer version', sources: ['raw_customers'] }, { name: 'fct_orders', pattern: 'dimensional', grain: 'one row per order', sources: ['raw_orders'] }, ], pipelines: [{ name: 'orders_elt', mode: 'batch', schedule: '0 * * * *', models: ['dim_customer', 'fct_orders'] }], contractFields: [ { field: 'order_id', type: 'string', classification: 'internal', semantics: 'natural order key' }, { field: 'customer_email', type: 'string', classification: 'pii', semantics: 'contact address' }, ], servingSurfaces: [{ name: 'orders-exec-dashboard', kind: 'dashboard' }], sla: { freshnessMinutes: 60, availabilityTarget: 0.999, completenessTarget: 0.995 }, backfill: { requested: true, window: '2024-01-01/2024-06-30', destructive: true }, cutoverStrategy: 'blue-green', }); Hard rules: Style-A agent tasks ONLY (zero kind:'shell' anywhere); every gate, verification and executed-run outputSchema declares evidence { type: 'array', minItems: 1 }; NO fallbacks — unknown tier, unknown model pattern, unknown pipeline mode, unknown serving-surface kind, unknown field classification, unknown cutover strategy, unknown test family, unknown SLA severity and never-raised routing lookups all THROW naming the source; a source that cannot be profiled FAILS the run rather than proceeding on an assumed schema; a failing test family or a failed adversarial gate BLOCKS cutover and the cutover gate is never raised; the backfill/cutover/deprecation executors are guarded solely by gate.approved === true and each has exactly ONE call site; no breakpoint in this process carries autoApproveAfterN; the timeline is accumulated in the ORCHESTRATOR only.

usesAgents