Airflow and Dagster Can't Backfill ML Experiments Right

Airflow and Dagster both partition work along a single axis of time. However, ML workflows require backfills along two axes - the time window of the data and the version of the code or config that processed it. Neither tool's partition model is built for this requirement, leading to duplicated DAGs and broken lineage. The issue arises because the orchestrator was designed with the assumption that the only thing changing between runs is the slice of data being processed, not the logic being applied to that data.

In Airflow, partitions are implicitly modeled as execution dates, while in Dagster, they are explicitly modeled but still keyed on date or business-time dimensions. This one-dimensional assumption holds for simple data warehouse loads but breaks when trying to re-run a model with a new feature logic over a range of time. The solution requires an orchestrator that can track code or config version as an explicit dimension, allowing backfills to be scoped by either axis independently or both together.

This would enable teams to easily compare outputs of different code versions and time windows, and avoid the current workaround of creating separate DAGs for each combination.