We spent thirteen weeks about to buy a bigger database
Our service dashboard had a panel called database time, and for thirteen weeks it showed a p99 of about nine hundred milliseconds at peak. We tuned queries, added two indexes, rewrote a join, and none of it moved the line. There was a quote on my desk for an instance upgrade at two thousand four hundred pounds a month and I was a day or two from approving it. What stopped me was a number that…
The article discusses a case of misattributed database performance issues at a company that spent thirteen weeks attempting to improve query times. Despite various optimizations, including adding indexes and rewriting joins, the p99 database time remained at nine hundred milliseconds. The issue stemmed from the instrumentation, which was incorrectly attributing connection wait times to database query duration.
The root cause was traced back to a single endpoint that opened a transaction, called an HTTP rendering service, and held onto the connection for up to eight seconds while waiting for a PDF. This accounted for forty calls per minute, significantly impacting the perceived performance of other queries. The solution involved refactoring the instrumentation to separate connection acquisition and query execution into distinct spans, providing a clearer picture of the system's performance.
Brief written by urgent.news from Dev.to's own syndicated text. Machine-written — may contain errors; check the original before relying on it.
