| Service | Purpose | Key Highlights |
|---|---|---|
| Azure Data Factory (ADF) | Primary Data Orchestrator | The powerhouse for orchestrating ingestion, transformation, and loading across hybrid and cloud data systems. Equivalent to Step Functions + MWAA combined. Supports dependency chaining, conditional logic, triggers, parameters, and CI/CD integration. |
| Azure Synapse / Fabric Pipelines | Lightweight Integrated Orchestration | Fabric reuses Synapse Pipelines for notebook/job orchestration within the SaaS ecosystem - tightly coupled with OneLake and Power BI. |
| Azure Logic Apps | Serverless Event-driven Workflow | Similar to Step Functions - used for low-code automation, API chaining, and event-based triggers (email alerts, Teams notifications, webhooks). |
| Azure Monitor | Centralized Observability Platform | Aggregates metrics, logs, and traces from all Azure services. Connects seamlessly with Log Analytics, Application Insights, and Power BI dashboards. |
| Azure Log Analytics / Application Insights | Log Analysis & Job Monitoring | Collects ETL logs, performance metrics, and custom telemetry for end-to-end visibility of data pipelines. |
| Azure Purview (for lineage) | Governance & Impact Analysis | Tracks lineage of datasets across ADF, Databricks, and Fabric Pipelines - similar to AWS CloudTrail + Glue Catalog lineage. |
Created
November 4, 2025 13:01
-
-
Save Anshuman-02905/11fe5202070b1dd5e84d26064323a1ce to your computer and use it in GitHub Desktop.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment