Urgent.News

What's breaking now, across thousands of outlets.

Tech

OpenTelemetry Collector to Loki Without Losing Trace Context

Originally published on kuryzhev.cloud The scenario A checkout request fails, you find the error in Grafana, and the log line has no trace link. The trace exists in Tempo, but nothing connects the two, so you copy IDs by hand. This is a common outcome when you ship logs from the OpenTelemetry Collector to Loki through a path that flattens the log record into a plain string and drops its trace…

A failing checkout request may leave the error in Grafana without any connection to the related trace in Tempo. This often happens when logs are sent from the OpenTelemetry Collector to Loki, resulting in the loss of trace context. The scenario is typical when using the deprecated Loki exporter in the OpenTelemetry Collector. To improve this situation, follow these steps:

1. Preserve trace context at the source: Ensure that the trace and span IDs are sent as structured metadata in the log record by using an OpenTelemetry SDK log bridge or appender. This way, the trace context remains intact and is not lost in the logging process.

2. Configure the OpenTelemetry Collector pipeline: Use the documented provisioning fields in the collector configuration, not features specific to a particular release. The configuration should include OTLP in as the data source, filelog receiver for Kubernetes container logs, and trace_parser operator to promote trace and span IDs into the record. Processors like memory_limiter, k8sattributes, and batch should be used to manage data flow, while the exporters should be set to Loki over native OTLP.

3. Avoid labeling trace context: Loki's OTLP ingestion automatically converts default resource attributes into index labels, such as service.name, k8s.namespace.name, and k8s.pod.name. However, trace_id and span_id should not be promoted to labels, as this would create a performance and cost issue. Only include the trace_id and span_id as structured metadata to maintain correlation with Tempo.

By following these steps and ensuring a proper configuration, you can connect logs from the OpenTelemetry Collector to Loki while preserving trace context, preventing the loss of valuable trace information.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

More from Wednesday 18 November →