workflow:data-quality-monitoring
Data Quality Monitoring overview
Continuous single-team workflow that proactively detects and resolves data quality issues before they propagate to downstream consumers, dashboards, or ML models. Automated data quality checks run after each pipeline execution, evaluating freshness, completeness, uniqueness, referential integrity, and distribution drift against defined expectations. Failures trigger alerts routed to the owning data engineer's on-call rotation. The engineer investigates root causes — upstream source changes, schema drift, or pipeline bugs — applies fixes, and documents the incident. Quality metrics and SLA compliance are reported weekly to data consumers. The workflow builds trust in the data platform and reduces costly downstream errors caused by silent data corruption.
Attributes
Outgoing edges
- domain:data-engineering·DomainData Engineering
- domain:databases·DomainDatabases
- domain:observability·DomainObservability
- role:data-engineer·RoleData Engineer
- role:analytics-engineer·RoleAnalytics Engineer
- role:data-analyst·RoleData Analyst
- role:bi-developer·RoleBI Developer
- responsibility:data-quality·ResponsibilityData Quality
- responsibility:on-call·ResponsibilityOn-Call
- responsibility:metric-definition·ResponsibilityMetric Definition
- responsibility:sla-management·ResponsibilitySLA Management
Incoming edges
- stack-profile:document-processing-pipeline·StackProfileDocument Processing Pipeline (OCR + NLP + Python + Elasticsearch + FastAPI)
- stack-profile:data-quality-governance·StackProfileData Quality / Governance Stack (Great Expectations, dbt, Airflow, PostgreSQL, Python)
- stack-profile:master-data-management·StackProfileMaster Data Management (Python, PostgreSQL, RabbitMQ, Airflow, FastAPI)
- lib-agent:data-engineering-analytics--data-governance-steward·LibraryAgentData Governance Steward Agent
- lib-agent:data-engineering-analytics--data-quality-engineer·LibraryAgentdata-quality-engineer
- lib-agent:data-science-ml--data-engineer·LibraryAgentdata-engineer
- lib-agent:data-science-ml--drift-detective·LibraryAgentdrift-detective
- lib-agent:data-science-ml--eda-analyst·LibraryAgenteda-analyst
- lib-agent:data-science-ml--incident-responder·LibraryAgentincident-responder
- lib-agent:data-science-ml--retraining-orchestrator·LibraryAgentretraining-orchestrator
- lib-skill:data-engineering-analytics--airflow-dag-analyzer·LibrarySkillairflow-dag-analyzer
- lib-skill:data-engineering-analytics--data-catalog-enricher·LibrarySkillData Catalog Enricher
- lib-skill:data-engineering-analytics--data-lineage-mapper·LibrarySkilldata-lineage-mapper
- lib-skill:data-engineering-analytics--data-quality-profiler·LibrarySkilldata-quality-profiler
- lib-skill:data-engineering-analytics--great-expectations-generator·LibrarySkillGreat Expectations Generator
- lib-skill:data-science-ml--arize-observability·LibrarySkillarize-observability
- lib-skill:data-science-ml--evidently-drift-detector·LibrarySkillevidently-drift-detector
- lib-skill:data-science-ml--great-expectations-validator·LibrarySkillgreat-expectations-validator
- lib-skill:data-science-ml--pandas-dataframe-analyzer·LibrarySkillpandas-dataframe-analyzer
- lib-skill:data-science-ml--whylabs-monitor·LibrarySkillwhylabs-monitor