II.
LibraryProcess overview
Reference · livelib-process:data-engineering-analytics--streaming-pipeline
streaming-pipeline overview
Streaming Data Pipeline Setup - Complete workflow for designing and implementing production-ready streaming data pipelines with Kafka/Kinesis setup, stream processing frameworks, windowing, state management, and comprehensive monitoring.
Attributes
displayName
streaming-pipeline
description
Streaming Data Pipeline Setup - Complete workflow for designing and implementing
production-ready streaming data pipelines with Kafka/Kinesis setup, stream processing frameworks,
windowing, state management, and comprehensive monitoring.
libraryPath
library/specializations/data-engineering-analytics/streaming-pipeline.js
specialization
data-engineering-analytics
references
- - Apache Kafka Documentation: https://kafka.apache.org/documentation/ - AWS Kinesis: https://docs.aws.amazon.com/kinesis/ - Apache Flink: https://flink.apache.org/ - Apache Spark Streaming: https://spark.apache.org/streaming/ - Kafka Streams: https://kafka.apache.org/documentation/streams/ - Stream Processing Patterns: https://www.confluent.io/blog/event-streaming-patterns/ - State Management in Flink: https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/state/ - Exactly-Once Semantics: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
example
const result = await orchestrate('specializations/data-engineering-analytics/streaming-pipeline', {
projectName: 'Real-time Analytics Pipeline',
streamingPlatform: 'kafka',
processingFramework: 'flink',
requirements: {
throughput: '100000 events/sec',
latency: 'sub-second',
dataRetention: '7 days',
stateBackend: 'rocksdb',
monitoring: true,
schemas: true,
exactlyOnce: true
}
});
usesAgents
- streaming-architect
- messaging-engineer
- schema-engineer
- processing-engineer
- state-management-specialist
- windowing-specialist
- connector-engineer
- sink-engineer
- backpressure-specialist
- monitoring-engineer
- lag-monitoring-specialist
- performance-engineer
- alerting-engineer
- autoscaling-engineer
- dr-specialist
- pipeline-validator
- technical-writer
Outgoing edges
lib_applies_to_domain1
- domain:data-engineering·DomainData Engineering
lib_belongs_to_specialization1
- specialization:data-engineering-analytics·Specialization
lib_implements_workflow1
- workflow:data-pipeline-deployment·WorkflowData Pipeline Deployment
uses_agent3
- lib-agent:software-architecture--performance-engineer·LibraryAgentperformance-engineer
- lib-agent:devops-sre-platform--dr-specialist·LibraryAgentdr-specialist
- lib-agent:meta--technical-writer·LibraryAgenttechnical-writer
Incoming edges
None.