II.
LibraryProcess overview
Reference · livelib-process:data-engineering-analytics--data-catalog
data-catalog overview
Comprehensive data catalog setup covering metadata management, platform selection (DataHub/Amundsen/Alation), data discovery, lineage visualization, and business glossary
Attributes
displayName
data-catalog
description
Comprehensive data catalog setup covering metadata management, platform selection (DataHub/Amundsen/Alation), data discovery, lineage visualization, and business glossary
libraryPath
library/specializations/data-engineering-analytics/data-catalog.js
specialization
data-engineering-analytics
references
- - DataHub: https://datahubproject.io/ - Amundsen: https://www.amundsen.io/ - Alation: https://www.alation.com/ - Apache Atlas: https://atlas.apache.org/ - Data Catalog Best Practices: https://cloud.google.com/architecture/data-catalog-best-practices
example
const result = await orchestrate('data-engineering-analytics/data-catalog', {
platform: 'datahub', // 'datahub', 'amundsen', 'alation', or null for evaluation
dataSources: [
{ type: 'snowflake', connection: 'account.region.snowflakecomputing.com', databases: ['analytics', 'raw'] },
{ type: 'postgres', host: 'localhost', databases: ['app_db'] },
{ type: 's3', bucket: 'data-lake', prefix: 'datasets/' },
{ type: 'kafka', brokers: ['kafka1:9092'], topics: ['events'] }
],
requirements: {
users: 50,
dataSets: 1000,
searchVolume: 'high',
integrations: ['tableau', 'looker', 'dbt', 'airflow'],
authentication: 'sso',
complianceRequirements: ['gdpr', 'ccpa']
},
enableLineage: true,
enableGlossary: true,
cloudProvider: 'aws'
});
usesAgents
- requirements-analyst
- platform-evaluator
- data-architect
- data-engineer
- metadata-architect
- lineage-engineer
- glossary-specialist
- search-engineer
- security-architect
- quality-engineer
- devops-engineer
- integration-engineer
- sre-engineer
- technical-writer
- project-manager
Outgoing edges
lib_applies_to_domain1
- domain:data-engineering·DomainData Engineering
lib_belongs_to_specialization1
- specialization:data-engineering-analytics·Specialization
lib_implements_workflow2
- workflow:data-pipeline-deployment·WorkflowData Pipeline Deployment
- workflow:data-governance-review·WorkflowData Governance Review
uses_agent5
- lib-agent:product-management--requirements-analyst·LibraryAgentRequirements Analyst
- lib-agent:software-architecture--data-architect·LibraryAgentdata-architect
- lib-agent:data-science-ml--data-engineer·LibraryAgentdata-engineer
- lib-agent:software-architecture--security-architect·LibraryAgentsecurity-architect
- lib-agent:meta--technical-writer·LibraryAgenttechnical-writer
Incoming edges
None.