How to integrate Apache Airflow with OpenLineage for end-to-end traceability

Integrating Apache Airflow with OpenLineage enables end-to-end traceability of your data pipelines. By following this guide, you can ensure that every DAG run in your Airflow instance emits structured lineage events, identifying exactly which tables each task read and wrote. This eliminates the need for manual documentation or stale lineage spreadsheets, as the orchestrator reports what it actually did during execution.

Before proceeding, ensure you meet the prerequisites: Apache Airflow 2.11.0 or later, Python 3.9 to 3.12, Docker for a local lineage backend, and a Postgres connection in Airflow (postgres_default) for SQL examples. The two key packages involved are the Airflow OpenLineage provider, which extracts Airflow metadata and transforms it into events, and openlineage-python, which transmits these events.

The provider can be upgraded independently of your Airflow version, allowing you to update transport fixes without altering your Airflow version.

The event model consists of three objects: Job, Run, and Dataset. Jobs represent your DAGs and individual tasks, while Runs denote one execution of a job with a unique run ID. Datasets are defined by a namespace and name, with facets containing atomic blocks of metadata such as schema, SQL text, column-level lineage, run state, and custom fields. Events are emitted on state transitions, including START, RUNNING, COMPLETE, FAIL, ABORT, and OTHER.

Common issues that hinder lineage graph construction include missing dataset identity, which is crucial for joining datasets across runs and forming a complete graph. To install the OpenLineage provider, use `pip install apache-airflow-providers-openlineage`, checking if the official Airflow Docker images already include it. The provider stays silent until configured with a transport destination.

Initially, use the console transport to write events to task logs and verify extraction works. After confirming, move to a real backend like Marquez, the reference implementation of the OpenLineage standard. Marquez provides a lineage UI and can be set up by cloning its repository, running the Docker setup script, and adjusting port configurations on macOS if necessary.

When configuring the transport, specify the URL, endpoint, and namespace to logically separate producers, preventing different Airflow installations from merging into a single graph. The namespace is essential for distinguishing between staging and production Airflows. Credentials for the transport should be stored in an Airflow connection ID extra rather than in airflow.cfg for better security.

Running a DAG with SQL operators is an effective way to trigger lineage events. SQL operators automatically derive inputs, outputs, and column-level relationships without manual configuration. Create a DAG that executes a SQL query, such as building daily order statistics from raw orders and customers tables. The provider will automatically generate lineage events connecting the input tables to the output table.

After running the DAG, verify the lineage graph by checking the visual representation at http://localhost:3000, ensuring the source tables connect to the output table and identifying which tasks are reporting their lineage. If certain tasks are silent, review their configurations, as EmptyOperator tasks may not emit events by default. Adding callbacks or task outlets can help represent these tasks in the lineage graph.

By following these steps, you can integrate Apache Airflow with OpenLineage to achieve comprehensive traceability of your data workflows, from task execution to downstream dependencies, without relying on manual documentation or outdated lineage records.