II.
LibraryProcess overview
Reference · livelib-process:data-engineering-analytics--data-quality-framework
data-quality-framework overview
Implement comprehensive data quality framework with dimensions, validation rules, monitoring, alerting, anomaly detection, and data profiling
Attributes
displayName
data-quality-framework
description
Implement comprehensive data quality framework with dimensions, validation rules, monitoring, alerting, anomaly detection, and data profiling
libraryPath
library/specializations/data-engineering-analytics/data-quality-framework.js
specialization
data-engineering-analytics
references
- - Great Expectations: https://greatexpectations.io/ - dbt Tests: https://docs.getdbt.com/docs/build/tests - Data Quality Dimensions: https://www.dataversity.net/six-key-dimensions-data-quality/ - Anomaly Detection: https://scikit-learn.org/stable/modules/outlier_detection.html - Data Profiling: https://github.com/ydataai/ydata-profiling
example
const result = await orchestrate('data-engineering-analytics/data-quality-framework', {
dataSources: [
{ type: 'database', connection: 'postgres://...', tables: ['users', 'orders'] },
{ type: 'file', path: 'data/transactions.parquet' }
],
qualityDimensions: ['accuracy', 'completeness', 'consistency', 'validity', 'timeliness', 'uniqueness'],
validationRules: {
accuracy: ['no_outliers_beyond_3_std', 'reference_data_match'],
completeness: ['required_fields_present', 'no_null_in_critical_columns'],
consistency: ['cross_table_integrity', 'format_consistency'],
validity: ['email_format', 'positive_amounts', 'valid_date_ranges'],
timeliness: ['data_age_within_24h', 'no_future_dates'],
uniqueness: ['primary_key_unique', 'no_duplicate_records']
},
thresholds: {
critical: 95,
warning: 85,
minimum: 70
},
monitoringEnabled: true,
alertingConfig: {
channels: ['email', 'slack'],
recipients: ['data-team@company.com'],
severityLevels: ['critical', 'high']
},
profilingEnabled: true
});
usesAgents
- general-purpose
Outgoing edges
lib_applies_to_domain1
- domain:data-engineering·DomainData Engineering
lib_belongs_to_specialization1
- specialization:data-engineering-analytics·Specialization
lib_implements_workflow1
- workflow:code-review·Workflow
Incoming edges
None.