II.
SkillArea overview
Reference · liveskill-area:etl-pipelines
ETL Pipelines overview
Designing extract-transform-load pipelines — idempotency, backfills, schema drift handling, and operational SLAs.
Attributes
displayName
ETL Pipelines
description
Designing extract-transform-load pipelines — idempotency,
backfills, schema drift handling, and operational SLAs.
domains
expertiseLevels
- intermediate
- expert
Outgoing edges
applies_to1
- specialization:data-engineering-analytics·Specialization
uses_stack_part1
- stack-part:scheduler·StackPartJob Scheduler / Orchestrator
Incoming edges
lib_requires_skill_area12
- lib-agent:code-migration-modernization--data-integrity-validator·LibraryAgentdata-integrity-validator
- lib-agent:data-engineering-analytics--data-orchestration-engineer·LibraryAgentData Orchestration Engineer Agent
- lib-agent:data-engineering-analytics--migration-specialist·LibraryAgentMigration Specialist Agent
- lib-agent:travel--python-etl-engineer·LibraryAgentpython-etl-engineer
- lib-agent:travel--sql-query-composer·LibraryAgentsql-query-composer
- lib-agent:travel--sqlite-schema-architect·LibraryAgentsqlite-schema-architect
- lib-agent:healthcare--clinical-informatics-specialist·LibraryAgentclinical-informatics-specialist
- lib-skill:code-migration-modernization--etl-pipeline-builder·LibrarySkilletl-pipeline-builder
- lib-skill:data-engineering-analytics--airflow-dag-analyzer·LibrarySkillairflow-dag-analyzer
- lib-skill:data-engineering-analytics--cdc-pattern-implementer·LibrarySkillCDC Pattern Implementer
- lib-skill:data-engineering-analytics--incremental-model-strategy-selector·LibrarySkillIncremental Model Strategy Selector
- lib-skill:healthcare--health-data-integration·LibrarySkillhealth-data-integration
prerequisite_for_learning1
- skill-area:data-analysis·SkillAreaData Analysis
requires_expertise4
- responsibility:data-pipeline-reliability·ResponsibilityData pipeline reliability
- role:data-engineer·RoleData Engineer
- role:data-quality-engineer·RoleData Quality Engineer
- role:etl-developer·RoleETL Developer
requires_skill_area12
- skill-area:etl-testing·SkillAreaETL Testing
- stack-profile:data-lakehouse·StackProfileData Lakehouse Stack (Databricks, Spark, Delta Lake, dbt, Airflow)
- stack-profile:batch-processing·StackProfileBatch Processing (Airflow + dbt + PostgreSQL + Python + S3)
- stack-profile:data-warehouse-bi·StackProfileData Warehouse / BI Stack (dbt, BigQuery, Metabase/Looker, Python, Airflow)
- stack-profile:data-quality-governance·StackProfileData Quality / Governance Stack (Great Expectations, dbt, Airflow, PostgreSQL, Python)
- stack-profile:master-data-management·StackProfileMaster Data Management (Python, PostgreSQL, RabbitMQ, Airflow, FastAPI)
- stack-profile:data-pipeline-orchestration·StackProfileData Pipeline Orchestration (Python, Airflow, dbt, PostgreSQL, Docker)
- stack-profile:etl-reverse-etl·StackProfileETL / Reverse ETL (Python, Airbyte, dbt, PostgreSQL, Airflow)
- stack-profile:data-lake-stack·StackProfileData Lake Stack (Spark, Object Storage, Delta/Iceberg)
- workflow:etl-pipeline-cost-optimization·WorkflowETL Pipeline Cost Optimization
- workflow:data-quality-check·WorkflowData Quality Check
- workflow:data-migration·WorkflowData Migration
stack_part_used_by1
- stack-part:scheduler·StackPartJob Scheduler / Orchestrator
used_for5
- tool:airbyte·ToolAirbyte
- tool:fivetran·ToolFivetran
- tool:dagster·ToolDagster
- tool:prefect·ToolPrefect
- tool:apache-beam·ToolApache Beam