Atlassian Rebuilds Metrics Pipeline Around OpenTelemetry at Massive Scale

Atlassian has recently shared its experience in migrating its metrics pipeline to OpenTelemetry at a massive scale. The company aimed to replace gostatsd, an internal metrics platform, with a system that could handle traces, logs, and the increasing number of services producing OpenTelemetry data. Instead of abruptly switching to OpenTelemetry, Atlassian retained the existing StatsD-over-UDP interface and rebuilt the pipeline behind it, turning the organization-wide migration into a platform-team effort.

They introduced purpose-built Collector distributions for collection, ingestion, aggregation, and forwarding, which allowed them to change one part of the system without replacing the others. By dividing the platform into four stages, Atlassian reduced sidecar cost by around 30% across its fleet and improved CPU distribution, reduced hot shards, and enhanced off-peak scaling.

The aggregation processor, a significant contributor to data volume, was also developed and open-sourced by Atlassian, cutting its CPU usage by approximately half. The migration process involved gradual rollouts, starting from development and staging workloads, and focused on preserving familiar operational workflows while old and new systems coexisted.

Atlassian aims to eventually replace the bespoke forwarding service with a stateless Collector distribution called metrics-gateway, which can fan out data to various destinations like SignalFx and S3. This approach differs from Airbnb's recent migration, where they used dual emission and VictoriaMetrics for streaming aggregation.