How to Tell Whether the Model Behind Your API Got Worse
A serving bug can break a few percent of a model's answers while every eval score you track stays flat. Counting the answers that fail a simple check catches it with thirty times less traffic. Engineer's view. This is one bad host behind a load balancer. One server in thirty returns garbage, and average latency barely moves. You find it by counting errors, not by averaging. A hosted model breaks…
To determine if the model powering your API has deteriorated, you need to count how many responses fail a simple test rather than relying on average evaluation scores. A single bad server behind a load balancer can break about a third of its model's answers, while the average latency and success rates may remain unchanged. To spot this, simply run a basic check on each response and tally the failures.
If that failure rate deviates significantly from its usual rate, it indicates a problem. For example, think of it as tasting a sip of lemonade from each cup your helper pours. If suddenly one out of thirty cups tastes salty, you know something is wrong. Similarly, ask your users a straightforward question after each interaction, like "Was this response helpful?"
Normal responses will yield no "yes" answers. If two "yes" replies appear on a good day, it signifies something went awry. The issue at hand is that one server behind your load balancer malfunctioned, returning errors roughly one in thirty requests. Your dashboard showed normal average latency and success rates, so the problem went unnoticed for days.
This scenario played out in August 2025 at Anthropic, where three serving bugs caused Claude's responses to degrade for about a month. One bug sent some requests to servers configured for a different context length, affecting 0.8% of requests initially and up to 16% at its peak. Another bug caused the model to occasionally insert irrelevant tokens, such as Thai characters in English replies.
Anthropic noted that their evaluations failed to capture this degradation, as the model could often recover from a single erroneous token, leading to minimal drops in quality scores. The solution is to count the responses that fail a quick check, compare that count to its normal rate, and monitor the traffic so that a genuine drop can be detected.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.