OpenPipeline routing: create a custom pipeline and route logs to it
All your logs take the same path, and the same processing rule applies to the payment service and to a test job. This tutorial shows OpenPipeline routing in practice: create a custom pipeline for one service, write the route that sends its logs there, and decide what happens to the rest of the stream. For an SRE or platform team, this is the starting point for controlling log volume, retention…
Log routing with OpenPipeline allows you to direct logs from a specific Kubernetes deployment to a custom pipeline for processing, while other logs continue on their original route. This tutorial explains how to set up this custom routing in Dynatrace SaaS.
Begin by examining existing logs to identify unique attributes that will help distinguish your target deployment, such as the deployment name and Kubernetes namespace. Use a query like `fetch logs | filter k8s.namespace.name == prod | summarize by k8s.deployment.name, count()` to determine the number of routes needed and what each one matches.
To create a custom pipeline, navigate to Settings > Process and contextualize > OpenPipeline, select the Logs configuration scope, and go to the Pipelines tab. Here, create a new pipeline with a name like "Checkout service pipeline" and add processors with matching conditions and processor-specific fields. Remember that no processors are needed at first, as the pipeline can be built later with masking or other processing rules.
Next, create a dynamic route by going to the dynamic routing tab and selecting Dynamic route. Provide a name, define a matching condition (e.g., `k8s.deployment.name == checkoutservice-*`), and select the custom pipeline you just created as the target. The wildcard `*` allows for variations in the deployment name.
Logs that don't match any new route will follow the Default route, which sends them to the existing Grail bucket. This approach allows you to gradually migrate services to custom pipelines without disrupting existing log handling. For logs that should be excluded from storage, use a condition that filters them out, such as `isNotNull(k8s.namespace.name) and k8s.namespace.name != prod`, and route them to a pipeline that uses the No storage assignment processor.
After saving your changes, wait a few minutes before checking if the routing works, as it only applies to newly ingested logs. Verify the routing by generating traffic or waiting for existing logs to be processed, then use a Notebook query like `fetch logs, from:now()-30m | filter matchesValue(k8s.deployment.name, checkoutservice-*) | summarize count()` to check if the intended logs are being routed correctly. If necessary, double-check the route condition, route position, and processor conditions.
When setting up multiple pipelines, follow best practices by assigning one pipeline per service or team, using explicit conditions to avoid "everything" matching, and ordering routes carefully to prevent conflicts. Additionally, version control both pipelines and routes using configuration as code and monitor ingestion with log ingestion monitoring. Permissions for pipelines include settings:objects:read, settings:objects:write for owners, and settings:objects:admin for administrators.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
