Building an Agent That Turns DataHub Metadata Into Airflow DAGs

A team developed an agent to automatically convert DataHub metadata into Airflow DAGs. While DataHub provides lineage, freshness SLAs, PII tags, and glossary terms, transforming that information into an Airflow DAG requires manual effort. The agent takes a table name and performs several actions: it traverses the lineage graph backwards to identify all upstream dependencies, reads tags and glossary terms related to the table, maps them to specific tasks (such as PII mapping to an audit task and freshness SLA to a freshness check), sorts the graph topologically, and reduces it before rendering a clean, deterministic Airflow 3 DAG.

The agent operates in two modes: an agent mode utilizing an LLM like Claude (or any model via OpenRouter) to explore the DataHub graph through the MCP protocol and generate a plan, and a script mode that bypasses the LLM for CI use. The developers made the decision that the LLM should not write code directly; instead, it outputs a structured plan that a separate deterministic renderer converts into actual Airflow task code.

This approach aims to strike a balance between the benefits of an agent and preventing the LLM from generating arbitrary code that could end up being scheduled in production.

The tool is still in a rough stage and lacks support for dbt or Spark operators, does not handle schema drift detection, and has only been tested with the DataHub NYC Taxi sample dataset. It can be found on GitHub at github.com/tdm291104/datahub-dag-generator and showcased on Devpost at devpost.com/software/datahub-dag-generator. A demo video is available on YouTube at youtube.com/watch?v=2f0uNNmSBfc. The creators welcome feedback and contributions, and any support, such as starring the repository, is appreciated.