II.
LibraryProcess overview
Reference · livelib-process:data-engineering-analytics--incremental-model
incremental-model overview
Incremental Model Setup - Design and implement incremental models for efficient large-scale data processing, covering incremental strategies (append-only, merge, delete+insert), unique keys, partitioning, clustering, backfill strategies, and performance optimization for data warehouse workloads.
Attributes
displayName
incremental-model
description
Incremental Model Setup - Design and implement incremental models for efficient large-scale data processing,
covering incremental strategies (append-only, merge, delete+insert), unique keys, partitioning, clustering, backfill strategies,
and performance optimization for data warehouse workloads.
libraryPath
library/specializations/data-engineering-analytics/incremental-model.js
specialization
data-engineering-analytics
references
- - dbt Incremental Models: https://docs.getdbt.com/docs/build/incremental-models - dbt Incremental Strategies: https://docs.getdbt.com/docs/build/incremental-strategy - Snowflake Incremental Models: https://docs.getdbt.com/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models - BigQuery Incremental Models: https://docs.getdbt.com/reference/resource-configs/bigquery-configs#merge-behavior-incremental-models - dbt Performance Best Practices: https://docs.getdbt.com/best-practices/how-we-build-our-metrics/3-optimize-performance
example
const result = await orchestrate('specializations/data-engineering-analytics/incremental-model', {
projectName: 'analytics_dbt',
dataWarehouse: 'Snowflake',
modelName: 'fct_orders',
sourceModel: 'stg_orders',
incrementalStrategy: 'merge',
updateFrequency: 'hourly',
dataVolume: 'large',
uniqueKey: ['order_id'],
partitionBy: { field: 'order_date', granularity: 'day' },
backfillRequired: true
});
usesAgents
- incremental-strategy-architect
- unique-key-engineer
- filter-logic-engineer
- partitioning-architect
- clustering-engineer
- model-developer
- test-engineer
- backfill-architect
- performance-optimizer
- monitoring-engineer
- validation-specialist
- documentation-writer
Outgoing edges
lib_applies_to_domain1
- domain:data-engineering·DomainData Engineering
lib_belongs_to_specialization1
- specialization:data-engineering-analytics·Specialization
lib_implements_workflow1
- workflow:data-pipeline-deployment·WorkflowData Pipeline Deployment
uses_agent2
- lib-agent:shared--test-engineer·LibraryAgenttest-engineer
- lib-agent:shared--documentation-writer·LibraryAgentdocumentation-writer
Incoming edges
None.