Urgent.News

What's breaking now, across thousands of outlets.

Tech

How to Tell Whether the Model Behind Your API Got Worse

A serving bug can break a few percent of a model's answers while every eval score you track stays flat. Counting the answers that fail a simple check catches it with thirty times less traffic. Engineer's view. This is one bad host behind a load balancer. One server in thirty returns garbage, and average latency barely moves. You find it by counting errors, not by averaging. A hosted model breaks…

To determine if the model powering your API has deteriorated, you need to count how many responses fail a simple test rather than relying on average evaluation scores. A single bad server behind a load balancer can break about a third of its model's answers, while the average latency and success rates may remain unchanged. To spot this, simply run a basic check on each response and tally the failures.

If that failure rate deviates significantly from its usual rate, it indicates a problem. For example, think of it as tasting a sip of lemonade from each cup your helper pours. If suddenly one out of thirty cups tastes salty, you know something is wrong. Similarly, ask your users a straightforward question after each interaction, like "Was this response helpful?"

Normal responses will yield no "yes" answers. If two "yes" replies appear on a good day, it signifies something went awry. The issue at hand is that one server behind your load balancer malfunctioned, returning errors roughly one in thirty requests. Your dashboard showed normal average latency and success rates, so the problem went unnoticed for days.

This scenario played out in August 2025 at Anthropic, where three serving bugs caused Claude's responses to degrade for about a month. One bug sent some requests to servers configured for a different context length, affecting 0.8% of requests initially and up to 16% at its peak. Another bug caused the model to occasionally insert irrelevant tokens, such as Thai characters in English replies.

Anthropic noted that their evaluations failed to capture this degradation, as the model could often recover from a single erroneous token, leading to minimal drops in quality scores. The solution is to count the responses that fail a quick check, compare that count to its normal rate, and monitor the traffic so that a genuine drop can be detected.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Technicolor CGM4140COM (Xfinity XB6) Login: 10.0.0.1 and the App-First Fix

If you are an Xfinity customer, the black tower sitting next to your TV is probably a Technicolor CGM4140COM — the gateway Xfinity ships as the XB6.

  • Access Technicolor CGM4140COM gateway admin page at 10.0.0.1
  • Use default username "admin" and password "password" to log in
  • App-first fix involves using Xfinity app for login issues

Separate decisions from context, and let agents propose but not ratify

When I build with AI coding agents, I keep running into the problems classical projects have. A lot of talk and notes, and no clear focus on what the decisions actually are. Who said it?

  • Smeldr separates context from reasoning and decision-making
  • Decisions proposed then ratified by authorized person
  • Ratified decisions become basis for further work

I wanted my PC in Home Assistant without an MQTT broker, so I wrote my own integration

I have a Snapdragon laptop, a desktop PC and a Home Assistant box on a Raspberry Pi. I wanted the computers in Home Assistant like any other device: sensors, power buttons, notifications, media.

  • Developer created Deskmate integration for Home Assistant without MQTT broker.
  • Deskmate uses encrypted WebSocket connection with X25519 key exchange per session.
  • Features include file sharing, screen time management, sensor monitoring, and power management.

More from Monday 16 November →