5,800 Data Engineers on What Is Actually in Production in 2026
Every year, someone publishes a "State of Data Engineering" survey with 200 respondents, mostly from their own Slack community, and calls it representative. I ignore most of them. When Astronomer dropped their State of Apache Airflow 2026 report with 5,818 respondents across 122 countries, I paid attention. That's actual sample size. I've spent years on both sides of the data engineering…
Every year, a State of Data Engineering survey is published with a small sample size of around 200 respondents, which is considered unrepresentative. Astronomer's 2026 State of Apache Airflow report, however, stands out with 5,818 respondents from 122 countries, a much larger and more diverse sample. The survey reveals several key insights about the current state of data engineering.
Firstly, Apache Airflow has gained significant traction, with over 3,600 unique contributors to the project, outpacing Spark and Kafka in terms of contributor count. This suggests that Airflow is becoming a more robust and collaborative open-source project, which bodes well for its long-term viability.
One of the most telling figures in the report is the migration status of teams using Airflow 2.x to the newer Airflow 3. Only 26% of respondents have completed the migration, while a whopping 84% are planning to do so. This highlights the challenges and costs associated with upgrading production DAGs to the newer version, especially given that Airflow 2.x will hit end-of-life in April 2026, leaving teams with no support or updates.
Moreover, the survey reveals a significant gap between general Airflow users and organizations that have been using Astronomer's managed platform for two or more years. While only 32% of all Airflow users report running GenAI or MLOps workloads in production, this figure jumps to 62% among Astronomer's managed-platform customers, and 83% among long-tenured platform customers.
This indicates that as organizations mature their data platforms and infrastructure, they become more capable of handling advanced workloads like GenAI and MLOps, a gap that could widen over time.
The survey also highlights the challenges of incorporating AI tools into data engineering workflows. While 51% of respondents use AI tools to write pipelines, many report issues such as hallucinations, lack of context, and outdated syntax. This suggests that although AI is becoming more prevalent in the industry, it is not yet a reliable replacement for experienced engineers who understand the nuances of data pipelines and can anticipate and prevent issues.
This gap underscores the ongoing need for human expertise in data engineering, even as AI technologies continue to evolve.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
