Our service got forty percent more expensive and nothing could say where

After a routine platform upgrade in July, the order service required approximately forty percent more CPU to perform the same tasks. The number of requests per second stayed the same, but the p95 (percentile) increased by sixty milliseconds, and the number of nodes rose by four. The changes were unnoticed, as no alerts were triggered.

It took three weeks for the team to uncover the reason behind the increased cost, despite having robust metrics, logs, and traces in place. These tools confirmed that the service had become more expensive, but they did not pinpoint the exact cause. The extra processing time was distributed evenly across all requests, indicating an issue within the process itself rather than at its boundaries.

The root cause was traced to thirty-four dependencies changing simultaneously. To identify the problematic dependency, the team performed a bisect operation. They created canary builds, reverted dependencies one by one, and compared the CPU seconds per request over an hour of real traffic. This process took five days, and it wasn't until continuous profiling that the issue was identified.

With the continuous profiler installed due to the exhaustion of other options, they discovered that thirty-one percent of the CPU time was spent on regular expression compilation inside a validation library. This library had stopped caching compiled patterns, unless a specific flag was provided.

The solution was a single line of configuration code. Now, CPU and heap profiles are continuously collected from every pod, costing just under two percent. These profiles are retained for thirty days and labeled by release, allowing for easy comparison of changes between different builds. Additionally, a heap dump is written to object storage when an out-of-memory situation occurs, as the container that could have revealed the issue is otherwise deleted within seconds.

Moreover, each release includes a panel displaying CPU seconds and bytes allocated per request, which now triggers a failure if the regression exceeds ten percent. This new monitoring system has detected two further regressions, both caused by the same issue. Ultimately, the team learned that while they had comprehensive monitoring in place, their existing tools did not provide insight into which specific code was consuming the extra resources.

This incident highlighted the need for a more targeted instrument to identify the root cause of cost increases.