Your Observability Bill Is a Codebase Nobody Reviews
It started with one $120K/year config fix. By the time I stopped asking questions, it was $150K+/year in waste — and a FinOps framework I wish I'd had from day one. Observability Spend Feels Like Insurance I'm an Sr. SRE at a company that processes payroll for hundreds of thousands of employees. Datadog is our nervous system — monitoring, APM, logs, synthetics, the works. Like every engineering…
This article recounts the author's experience with uncontrolled observability spend that ballooned to $150K+/year. Datadog, their monitoring platform, grew steadily as new services and teams were added. The author explains that observability spend feels like insurance - you don't optimise it because it's expected to grow with the codebase. However, the author argues that observability spend is a codebase that accumulates config, drifts and carries dead code, unlike actual code which gets regularly reviewed.
The author provides three examples of wasteful observability practices. Firstly, scanning all logs for PII data by default, regardless of where PII actually appears. This was fixed by restricting scanning to only the relevant log sources. Secondly, a shared library overriding the service name in instrumentation tags, creating ghost services with their own monitors and alerts.
The fix was to let each service own its service name at the edge configuration, not inside shared dependencies. Finally, staging load balancer access logs were being indexed in the production log index, consuming a quarter of the production quota. This was fixed by filtering the production index to only include production environment logs.
The author outlines an Observability FinOps Framework with four layers. Layer 1 is knowing what you pay for, which involves answering key questions about your observability platform like how many services exist, which log indexes contain which environment data, etc. Layer 2 is measuring each signal, meaning tracking metrics and costs for each observability signal.
Layer 3 is governing in the pipeline, where observability configurations should be audited and optimised as part of the dev workflow. Layer 4 is treating cost as an incident signal, so that when cost anomalies arise they trigger alerts and investigations like any other incident. The author wishes they had implemented this framework from the outset to proactively catch and prevent observability waste.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
