{
  "id": 15892495,
  "title": "Airflow and Dagster Can't Backfill ML Experiments Right",
  "url": "https://urgent.news/2026/10/21/airflow-and-dagster-cant-backfill-ml-experiments-right",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-21T13:16:03.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/aiexplore369zoho/airflow-and-dagster-cant-backfill-ml-experiments-right-4foj"
  },
  "original_language": "en",
  "account": "Airflow and Dagster both partition work along a single axis of time. However, ML workflows require backfills along two axes - the time window of the data and the version of the code or config that processed it. Neither tool's partition model is built for this requirement, leading to duplicated DAGs and broken lineage. The issue arises because the orchestrator was designed with the assumption that the only thing changing between runs is the slice of data being processed, not the logic being applied to that data.\n\nIn Airflow, partitions are implicitly modeled as execution dates, while in Dagster, they are explicitly modeled but still keyed on date or business-time dimensions. This one-dimensional assumption holds for simple data warehouse loads but breaks when trying to re-run a model with a new feature logic over a range of time. The solution requires an orchestrator that can track code or config version as an explicit dimension, allowing backfills to be scoped by either axis independently or both together. This would enable teams to easily compare outputs of different code versions and time windows, and avoid the current workaround of creating separate DAGs for each combination.",
  "summary": "TL;DR — Airflow and Dagster both partition work along a single axis: time. ML workflows need backfills along two axes at once — time window and code/config version — and neither tool's partition model was built for that. The result is a mess of duplicated DAGs and broken lineage that teams mistake for an organizational problem instead of an architectural one. Every mainstream orchestrator —…",
  "key_points": [
    "Airflow and Dagster partition work along a single time axis.",
    "ML workflows need backfills along two axes: data time window and code/config version.",
    "Current orchestrators can't track code/config version as explicit dimension."
  ],
  "editors_take": "Current orchestration architectures fail machine learning teams by treating logic as a constant, forcing manual workarounds that compromise data lineage and complicate the comparison of different model versions.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}