What It Really Takes to Run OpenTelemetry

OpenTelemetry emerged to solve a genuine issue in the industry - the need for a vendor-neutral method to collect traces, metrics, and logs. This open-source solution allows engineering teams to generate these telemetry data once and send it to any destination they choose. However, the conversation around OpenTelemetry adoption often overlooks the challenges that arise after the SDKs have been integrated.

Once the SDKs are in place, the framework is free, but running it effectively is not. Collectors, which are responsible for receiving and processing telemetry data, can become a significant infrastructure burden. Teams often find themselves running multiple collector instances, each tailored to a specific region or cluster. They must then tune various settings, such as batch and memory limits, and monitor the collectors for any performance bottlenecks.

This adds an additional layer of responsibility on top of the monitoring responsibilities that the teams were already handling.

Furthermore, storage and retention decisions fall under the team's purview. OTel defines how data is generated and transmitted, but it does not determine where it is stored. This means that teams must select and operate their own backend, typically a trace store, a time-series database for metrics, and a log index. They are then responsible for building queries and dashboards to make this data usable, and every upgrade across this chain requires coordination.

Cross-signal correlation, which involves linking traces, metrics, and logs together to diagnose issues, is another ongoing engineering task. While OTel provides a foundation for this, the actual correlation layer must be built and maintained continuously as schemas evolve. This is where the on-call burden shifts. With a managed platform, a vendor's SRE team would handle ingestion issues, but with a self-run OTel stack, it is usually the same engineers who are supposed to be using the telemetry to fix application problems who now have to maintain the pipeline that delivers it.

Another factor to consider is upgrade churn. OTel semantic conventions and SDKs evolve rapidly, requiring ongoing work to stay current across all instrumented services. This is particularly true in polyglot environments, where staying updated can consume significant engineering hours every quarter.

Despite these challenges, self-managed OTel is not necessarily a bad choice. It may be a viable option for teams with the platform engineering capacity and a clear need for full control over their telemetry pipeline. However, the honest accounting is that free software still incurs labor costs, and these costs can scale with the number of services, languages, and regions being instrumented.

The real trade-off lies in where the labor goes. With a managed backend, the operational weight of running collectors, tuning storage, and building correlation logic is shifted to the platform provider. This frees up engineering hours for other tasks, but it also means giving up some control. Conversely, self-hosting allows for deeper customization and control, but it comes with the added burden of maintaining the infrastructure and handling upgrades.

The decision ultimately depends on the team's platform engineering practices, compliance requirements, and the depth of Kubernetes native maturity. Regardless of the choice, the instrumentation layer using OTel remains a valuable investment. The question that remains is whether the underlying collector, storage, and correlation work is still the best use of the team's time.