II.
LibraryProcess overview
Reference · livelib-process:data-engineering-analytics--pipeline-migration
pipeline-migration overview
Data Pipeline Migration - Comprehensive workflow for migrating data pipelines with assessment, migration strategy, dual-run validation, cutover planning, and rollback procedures to ensure zero-downtime migrations.
Attributes
displayName
pipeline-migration
description
Data Pipeline Migration - Comprehensive workflow for migrating data pipelines with
assessment, migration strategy, dual-run validation, cutover planning, and rollback procedures
to ensure zero-downtime migrations.
libraryPath
library/specializations/data-engineering-analytics/pipeline-migration.js
specialization
data-engineering-analytics
references
- - Data Pipeline Migration Best Practices: https://www.dataengineeringweekly.com/ - Zero-Downtime Migration Patterns: https://martinfowler.com/bliki/StranglerFigApplication.html - Data Quality Validation: https://www.great-expectations.io/ - Dual-Run Testing: https://cloud.google.com/architecture/migration-to-gcp-getting-started - Pipeline Testing: https://docs.getdbt.com/docs/building-a-dbt-project/tests
example
const result = await orchestrate('specializations/data-engineering-analytics/pipeline-migration', {
projectName: 'Legacy ETL to Modern ELT Migration',
sourceSystem: {
type: 'legacy-etl',
platform: 'informatica',
dataVolume: '500GB daily'
},
targetSystem: {
type: 'modern-elt',
platform: 'dbt',
cloudProvider: 'aws'
},
requirements: {
zeroDo wntime: true,
dualRunDuration: '30 days',
dataQualityValidation: true,
performanceComparison: true,
rollbackStrategy: true
}
});
usesAgents
- migration-assessor
- dependency-analyzer
- migration-strategist
- pipeline-designer
- data-quality-engineer
- dual-run-engineer
- validation-engineer
- performance-analyzer
- cutover-planner
- rollback-strategist
- training-coordinator
- technical-writer
- monitoring-engineer
- cost-optimizer
Outgoing edges
lib_applies_to_domain1
- domain:data-engineering·DomainData Engineering
lib_belongs_to_specialization1
- specialization:data-engineering-analytics·Specialization
lib_implements_workflow2
- workflow:technical-debt-reduction·WorkflowTechnical Debt Reduction
- workflow:data-backfill-procedure·WorkflowData Backfill Procedure
uses_agent4
- lib-agent:software-architecture--migration-strategist·LibraryAgentmigration-strategist
- lib-agent:data-engineering-analytics--data-quality-engineer·LibraryAgentdata-quality-engineer
- lib-agent:meta--technical-writer·LibraryAgenttechnical-writer
- lib-agent:ai-agents-conversational--cost-optimizer·LibraryAgentcost-optimizer
Incoming edges
None.