II.
Domain overview
Reference · livedomain:data-engineering
Data Engineering overview
The Data Engineering domain — building and operating pipelines, batch and streaming ETL/ELT, data warehouses and lakes, schema evolution, data quality, and the orchestration tooling around them. Distinct from Data Science (analysis/modeling) and ML-Ops (model lifecycle).
Attributes
displayName
Data Engineering
description
The Data Engineering domain — building and operating pipelines, batch
and streaming ETL/ELT, data warehouses and lakes, schema evolution,
data quality, and the orchestration tooling around them. Distinct from
Data Science (analysis/modeling) and ML-Ops (model lifecycle).
Outgoing edges
contains6
- topic:event-mesh·TopicEvent Mesh
- topic:message-driven-architecture·TopicMessage-Driven Architecture
- topic:partitioning·TopicPartitioning
- topic:publish-subscribe·TopicPublish-Subscribe
- topic:publisher-subscriber-pattern·TopicPublisher/Subscriber (PubSub) Pattern
- topic:sharding·TopicSharding
Incoming edges
applies_to50
- skill-area:kafka-stream-processing·SkillAreaKafka Stream Processing
- skill-area:postgres-tuning·SkillAreaPostgres Performance Tuning
- skill-area:python-data-pipelines·SkillAreaPython Data Pipelines
- skill-area:data-collection·SkillAreaData Collection
- skill-area:test-data-management·SkillAreaTest Data Management
- skill-area:data-mesh-design·SkillAreaData Mesh Design
- skill-area:data-contract·SkillAreaData Contracts
- skill-area:data-catalog-management·SkillAreaData Catalog Management
- skill-area:real-time-analytics·SkillAreaReal-Time Analytics
- skill-area:stream-processing-design·SkillAreaStream Processing Design
- skill-area:change-data-capture·SkillAreaChange Data Capture
- skill-area:data-lakehouse-architecture·SkillAreaData Lakehouse Architecture
- skill-area:feature-engineering-production·SkillAreaProduction Feature Engineering
- skill-area:event-driven-architecture·SkillAreaEvent-Driven Architecture
- skill-area:graph-databases·SkillAreaGraph Databases
- stack-profile:data-lakehouse·StackProfileData Lakehouse Stack (Databricks, Spark, Delta Lake, dbt, Airflow)
- stack-profile:stream-processing·StackProfileStream Processing Stack (Kafka, Flink, Schema Registry, Prometheus)
- stack-profile:document-processing-pipeline·StackProfileDocument Processing Pipeline (OCR + NLP + Python + Elasticsearch + FastAPI)
- stack-profile:batch-processing·StackProfileBatch Processing (Airflow + dbt + PostgreSQL + Python + S3)
- stack-profile:adtech-real-time-bidding·StackProfileAdTech Real-Time Bidding Stack (Go, Redis, Kafka, ClickHouse, Prometheus)
- stack-profile:knowledge-graph-platform·StackProfileKnowledge Graph Platform (Neo4j, Python, FastAPI, React, D3, Elasticsearch)
- stack-profile:data-warehouse-bi·StackProfileData Warehouse / BI Stack (dbt, BigQuery, Metabase/Looker, Python, Airflow)
- stack-profile:data-quality-governance·StackProfileData Quality / Governance Stack (Great Expectations, dbt, Airflow, PostgreSQL, Python)
- stack-profile:master-data-management·StackProfileMaster Data Management (Python, PostgreSQL, RabbitMQ, Airflow, FastAPI)
- stack-profile:julia-data-service·StackProfileJulia Data Service (Julia, Python, PostgreSQL, Docker)
- stack-profile:data-pipeline-orchestration·StackProfileData Pipeline Orchestration (Python, Airflow, dbt, PostgreSQL, Docker)
- stack-profile:etl-reverse-etl·StackProfileETL / Reverse ETL (Python, Airbyte, dbt, PostgreSQL, Airflow)
- stack-profile:real-time-analytics-stack·StackProfileReal-Time Analytics Stack (Kafka, ClickHouse, Grafana, dbt)
- stack-profile:data-lake-stack·StackProfileData Lake Stack (Spark, Object Storage, Delta/Iceberg)
- stack-profile:event-driven-stack·StackProfileEvent-Driven Stack (Kafka, Consumers, Schema Registry)
- topic:message-driven-architecture·TopicMessage-Driven Architecture
- topic:publish-subscribe·TopicPublish-Subscribe
- topic:sharding·TopicSharding
- topic:partitioning·TopicPartitioning
- topic:event-mesh·TopicEvent Mesh
- topic:publisher-subscriber-pattern·TopicPublisher/Subscriber (PubSub) Pattern
- topic:event-driven-architecture·TopicEvent-Driven Architecture
- topic:database-sharding·TopicDatabase Sharding
- topic:data-mesh·TopicData Mesh
- role:streaming-engineer·RoleStreaming Engineer
- role:integration-engineer·RoleIntegration Engineer
- role:annotation-lead·RoleAnnotation Lead
- role:data-steward·RoleData Steward
- role:data-quality-engineer·RoleData Quality Engineer
- role:etl-developer·RoleETL Developer
- role:principal-data-engineer·RolePrincipal Data Engineer
- role:head-of-data·RoleHead of Data
- role:data-architect·RoleData Architect
- role:data-governance-lead·RoleData Governance Lead
- role:chief-data-officer·RoleChief Data Officer
applies_to_domain24
- workflow:data-pipeline-deployment·WorkflowData Pipeline Deployment
- workflow:data-quality-monitoring·WorkflowData Quality Monitoring
- workflow:data-governance-review·WorkflowData Governance Review
- workflow:schema-migration·WorkflowSchema Migration
- workflow:data-backfill-procedure·WorkflowData Backfill Procedure
- workflow:precision-agriculture-data-pipeline·WorkflowPrecision Agriculture Data Pipeline
- workflow:data-quality-scorecard-review·WorkflowData Quality Scorecard Review
- workflow:dashboard-development-cycle·WorkflowDashboard Development Cycle
- workflow:data-quality-investigation·WorkflowData Quality Investigation
- workflow:data-mesh-domain-ownership-review·WorkflowData Mesh Domain Ownership Review
- workflow:real-time-streaming-health-check·WorkflowReal-Time Streaming Health Check
- workflow:data-warehouse-schema-governance·WorkflowData Warehouse Schema Governance
- workflow:etl-pipeline-cost-optimization·WorkflowETL Pipeline Cost Optimization
- workflow:data-access-request-workflow·WorkflowData Access Request Workflow
- workflow:data-warehouse-cost-optimization·WorkflowData Warehouse Cost Optimization
- workflow:data-quality-check·WorkflowData Quality Check
- workflow:data-migration·WorkflowData Migration
- workflow:civic-data-publication·WorkflowCivic Data Publication
- workflow:clinical-trial-data-management·WorkflowClinical Trial Data Management
- workflow:resource-estimation-review·WorkflowResource Estimation Review
- workflow:pharmacovigilance-signal-detection·WorkflowPharmacovigilance Signal Detection
- workflow:alternative-data-evaluation·WorkflowAlternative Data Evaluation
- workflow:athlete-performance-analytics-review·WorkflowAthlete Performance Analytics Review
- workflow:market-data-feed-validation·WorkflowMarket Data Feed Validation
belongs_to_domain2
- topic:SQL-vs-NoSQL·TopicSQL vs. NoSQL
- topic:retrieval-augmented-generation-patterns·TopicRAG Patterns
lib_applies_to_domain63
- tool-server:mcp-azure-cosmos-db·ToolServerAzure Cosmos DB MCP Server
- tool-server:mcp-pyairbyte·ToolServerPyAirbyte MCP Server
- tool-server:mcp-fivetran·ToolServerFivetran MCP Server
- tool-server:mcp-dagster·ToolServerDagster MCP Server
- tool-server:mcp-prefect·ToolServerPrefect MCP Server
- tool-server:mcp-snowflake·ToolServerSnowflake MCP Server
- tool-server:mcp-bigquery·ToolServerBigQuery MCP Server
- tool-server:mcp-google-sheets·ToolServerGoogle Sheets MCP Server
- tool-server:mcp-rabbitmq·ToolServerRabbitMQ MCP Server
- tool-server:mcp-nats·ToolServerNATS MCP Server
- tool-server:mcp-kafka·ToolServerKafka MCP Server
- tool-server:mcp-segment·ToolServerSegment MCP Server
- lib-agent:data-engineering-analytics--bi-analytics-engineer·LibraryAgentBI Analytics Engineer Agent
- lib-agent:data-engineering-analytics--data-governance-steward·LibraryAgentData Governance Steward Agent
- lib-agent:data-engineering-analytics--data-orchestration-engineer·LibraryAgentData Orchestration Engineer Agent
- lib-agent:data-engineering-analytics--data-quality-engineer·LibraryAgentdata-quality-engineer
- lib-agent:data-engineering-analytics--data-warehouse-architect·LibraryAgentdata-warehouse-architect
- lib-agent:data-engineering-analytics--dbt-project-engineer·LibraryAgentdbt-project-engineer
- lib-agent:data-engineering-analytics--dimensional-modeler·LibraryAgentDimensional Modeler Agent
- lib-agent:data-engineering-analytics--migration-specialist·LibraryAgentMigration Specialist Agent
- lib-agent:data-engineering-analytics--ml-feature-engineer·LibraryAgentML Feature Engineer Agent
- lib-agent:data-engineering-analytics--streaming-pipeline-engineer·LibraryAgentStreaming Pipeline Engineer Agent
- lib-process:data-engineering-analytics--ab-testing-pipeline·LibraryProcessab-testing-pipeline
- lib-process:data-engineering-analytics--bi-dashboard·LibraryProcessbi-dashboard
- lib-process:data-engineering-analytics--data-catalog·LibraryProcessdata-catalog
- lib-process:data-engineering-analytics--data-lineage·LibraryProcessdata-lineage
- lib-process:data-engineering-analytics--data-quality-framework·LibraryProcessdata-quality-framework
- lib-process:data-engineering-analytics--data-warehouse-setup·LibraryProcessdata-warehouse-setup
- lib-process:data-engineering-analytics--dbt-model-development·LibraryProcessdbt-model-development
- lib-process:data-engineering-analytics--dbt-project-setup·LibraryProcessdbt-project-setup
- lib-process:data-engineering-analytics--dimensional-model·LibraryProcessdimensional-model
- lib-process:data-engineering-analytics--etl-elt-pipeline·LibraryProcessetl-elt-pipeline
- lib-process:data-engineering-analytics--feature-store·LibraryProcessfeature-store
- lib-process:data-engineering-analytics--incremental-model·LibraryProcessincremental-model
- lib-process:data-engineering-analytics--metrics-layer·LibraryProcessmetrics-layer
- lib-process:data-engineering-analytics--obt-creation·LibraryProcessobt-creation
- lib-process:data-engineering-analytics--pipeline-migration·LibraryProcesspipeline-migration
- lib-process:data-engineering-analytics--query-optimization·LibraryProcessquery-optimization
- lib-process:data-engineering-analytics--scd-implementation·LibraryProcessscd-implementation
- lib-process:data-engineering-analytics--streaming-pipeline·LibraryProcessstreaming-pipeline
- lib-skill:data-engineering-analytics--ab-test-statistical-analyzer·LibrarySkillA/B Test Statistical Analyzer
- lib-skill:data-engineering-analytics--airflow-dag-analyzer·LibrarySkillairflow-dag-analyzer
- lib-skill:data-engineering-analytics--apache-spark-optimizer·LibrarySkillApache Spark Optimizer
- lib-skill:data-engineering-analytics--batch-vs-stream-tradeoffs·LibrarySkillbatch-vs-stream-tradeoffs
- lib-skill:data-engineering-analytics--bi-semantic-layer-generator·LibrarySkillBI Semantic Layer Generator
- lib-skill:data-engineering-analytics--cdc-pattern-implementer·LibrarySkillCDC Pattern Implementer
- lib-skill:data-engineering-analytics--cost-optimizer·LibrarySkillCost Optimizer (Cloud Data Platforms)
- lib-skill:data-engineering-analytics--data-catalog-enricher·LibrarySkillData Catalog Enricher
- lib-skill:data-engineering-analytics--data-lineage-mapper·LibrarySkilldata-lineage-mapper
- lib-skill:data-engineering-analytics--data-quality-profiler·LibrarySkilldata-quality-profiler
- lib-skill:data-engineering-analytics--dbt-project-analyzer·LibrarySkilldbt-project-analyzer
- lib-skill:data-engineering-analytics--dimensional-model-validator·LibrarySkillDimensional Model Validator
- lib-skill:data-engineering-analytics--etl-testing·LibrarySkilletl-testing
- lib-skill:data-engineering-analytics--feature-engineering-optimizer·LibrarySkillFeature Engineering Optimizer
- lib-skill:data-engineering-analytics--great-expectations-generator·LibrarySkillGreat Expectations Generator
- lib-skill:data-engineering-analytics--incremental-model-strategy-selector·LibrarySkillIncremental Model Strategy Selector
- lib-skill:data-engineering-analytics--kafka-topic-designer·LibrarySkillKafka Topic Designer
- lib-skill:data-engineering-analytics--obt-design-optimizer·LibrarySkillOBT Design Optimizer
- lib-skill:data-engineering-analytics--scd-implementation-generator·LibrarySkillSCD Implementation Generator
- lib-skill:data-engineering-analytics--schema-evolution-manager·LibrarySkillSchema Evolution Manager
- lib-skill:data-engineering-analytics--spark-jobs·LibrarySkillspark-jobs
- lib-skill:data-engineering-analytics--sql-query-optimizer·LibrarySkillsql-query-optimizer
- lib-skill:data-engineering-analytics--stream-processing-windowing-designer·LibrarySkillStream Processing Windowing Designer
requires_skill2
- role:analytics-engineer·RoleAnalytics Engineer
- role:data-engineer·RoleData Engineer