{
  "id": 9720677,
  "title": "Our service got forty percent more expensive and nothing could say where",
  "url": "https://urgent.news/2026/09/25/our-service-got-forty-percent-more-expensive-and-nothing-could-say",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-25T07:01:30.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sergey_shinder_ab2d943365/our-service-got-forty-percent-more-expensive-and-nothing-could-say-where-j2n"
  },
  "original_language": "en",
  "account": "After a routine platform upgrade in July, the order service required approximately forty percent more CPU to perform the same tasks. The number of requests per second stayed the same, but the p95 (percentile) increased by sixty milliseconds, and the number of nodes rose by four. The changes were unnoticed, as no alerts were triggered. It took three weeks for the team to uncover the reason behind the increased cost, despite having robust metrics, logs, and traces in place. These tools confirmed that the service had become more expensive, but they did not pinpoint the exact cause. The extra processing time was distributed evenly across all requests, indicating an issue within the process itself rather than at its boundaries.\n\nThe root cause was traced to thirty-four dependencies changing simultaneously. To identify the problematic dependency, the team performed a bisect operation. They created canary builds, reverted dependencies one by one, and compared the CPU seconds per request over an hour of real traffic. This process took five days, and it wasn't until continuous profiling that the issue was identified. With the continuous profiler installed due to the exhaustion of other options, they discovered that thirty-one percent of the CPU time was spent on regular expression compilation inside a validation library. This library had stopped caching compiled patterns, unless a specific flag was provided.\n\nThe solution was a single line of configuration code. Now, CPU and heap profiles are continuously collected from every pod, costing just under two percent. These profiles are retained for thirty days and labeled by release, allowing for easy comparison of changes between different builds. Additionally, a heap dump is written to object storage when an out-of-memory situation occurs, as the container that could have revealed the issue is otherwise deleted within seconds. Moreover, each release includes a panel displaying CPU seconds and bytes allocated per request, which now triggers a failure if the regression exceeds ten percent. This new monitoring system has detected two further regressions, both caused by the same issue. Ultimately, the team learned that while they had comprehensive monitoring in place, their existing tools did not provide insight into which specific code was consuming the extra resources. This incident highlighted the need for a more targeted instrument to identify the root cause of cost increases.",
  "summary": "After a routine platform bump in July, the order service needed about forty percent more CPU for the same work. Requests per second unchanged, p95 up by sixty milliseconds, node count up by four. Nothing was broken and nothing alerted. It took us three weeks to find out why, and the tooling we were so pleased with contributed almost nothing. We had metrics, logs and traces, and they agreed the…",
  "key_points": [
    "Service costs increased by forty percent after July upgrade",
    "CPU usage rose by forty percent, p95 latency increased by sixty milliseconds",
    "Root cause traced to thirty-four dependencies changing simultaneously"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}