Zero-Code OpenTelemetry Tracing for Dagster

Dagster pipelines can now benefit from automatic, zero-code tracing using the opentelemetry-instrumentation-dagster package. This new tool allows for distributed tracing without modifying any pipeline code. To use it, simply install the package, set environment variables for the OpenTelemetry service name and OTEL_EXPORTER_OTLP_ENDPOINT, and run the Dagster command through the opentelemetry-instrument launcher.

This triggers automatic tracing for all @op, @asset, @multi_asset, @asset_check, and @dbt_assets functions in the pipeline, without any explicit decorators or imports. The package achieves this by patching decorator factories, such as dagster.op, dagster.asset, and dagster.multi_asset, before Dagster builds the OpDefinition or AssetsDefinition.

This ensures that tracing occurs automatically, even when processes are spawned in a multi-process execution environment. The package works by inserting a sitecustomize.py file into the PYTHONPATH, which triggers the auto-instrumentation before your code imports the Dagster library. This allows for consistent tracing across all processes, including those spawned by the k8s_job_executor.

However, the @graph_asset decorator is intentionally excluded from this automatic tracing, as its function is a composition function called at definition time and does not receive a runtime context. The package has been thoroughly tested in real infrastructure, including a Kubernetes cluster with Postgres-backed run storage and k8s_job_executor, ensuring its reliability and effectiveness in tracing Dagster pipelines.