{
  "id": 8813902,
  "title": "Stop trying to make Airflow work for Medallion pipelines",
  "url": "https://urgent.news/2026/09/21/stop-trying-to-make-airflow-work-for-medallion-pipelines",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-21T01:21:03.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/aniketsoni/stop-trying-to-make-airflow-work-for-medallion-pipelines-15ge"
  },
  "original_language": "en",
  "account": "Stop trying to make Airflow work for Medallion pipelines. The concept of a universal orchestrator is a myth that keeps data engineers up at night. You may have been led to believe that picking the right tool—usually the one with the most GitHub stars—will magically transform your medallion architecture into a self-healing, high-performing pipeline. However, this is not the case.\n\nAirflow is not the right tool for every organization. It thrives in environments where a dedicated team manages the scheduler. However, for most others, it's a source of dependency hell and fragility. Using Airflow leads to 40% of your time spent managing the orchestrator rather than the data itself. In a medallion pipeline, where atomicity is crucial, Airflow fails to provide the necessary guarantees.\n\nAWS Step Functions are the ideal choice for AWS-based Medallion architectures. They offer a 'set and forget' experience without the need for server management. The key feature here is the 'Wait for Callback' functionality, which allows long-running jobs to pause until completion. However, be cautious of hitting execution history limits when processing large datasets.\n\nDatabricks Workflows are the native choice for Databricks-backed Medallion pipelines using Delta Lake. It handles cluster lifecycle, notebook execution, and Delta commits effectively. Unlike external tools, Workflows are aware of cluster status, ensuring efficient retries in case of node failures.\n\nAvoid creating a 'God DAG' with a single chain of Bronze, Silver, and Gold transformations. This monolithic approach creates a ticking time bomb. If Gold fails, you must restart the entire process, reprocessing Bronze and Silver again. Instead, use an event-driven model with decoupled triggers.\n\nThe orchestrator's database should not be the source of truth for data quality. Airflow variables to track processed timestamps often lead to issues with backfilling or re-running data. Instead, keep state in the data itself using a metadata table in your Delta Lake.\n\nFinally, don't measure success based on a green checkmark in the UI. Jobs may appear successful while loading empty data or corrupting partitions. Implement data quality checks using tools like dbt or Great Expectations. If these checks fail, the pipeline should stop before proceeding to the Gold layer. Decoupling logic and focusing on observability are key to building a robust, orchestrator-agnostic pipeline.",
  "summary": "The \"universal orchestrator\" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told that if you just pick the right tool—usually the one with the most GitHub stars—your medallion architecture will magically materialize into a self-healing, performant pipeline. I’ve spent six years cleaning up the aftermath of this delusion. I’ve seen…",
  "key_points": [],
  "editors_take": "Rethinking the role of orchestrators in Medallion pipelines means prioritizing data quality and atomicity over tool choice, and decoupling logic to build robust, self-healing architectures.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}