Urgent.News

What's breaking now, across thousands of outlets.

More in Tech

Your evals pass. That doesn't mean they work.

Last week I wrote about tracelint , a linter that catches structural bugs in agent traces — the classic one being an agent that calls charge_card , gets a failure back, and just...

  • Muteval tool identifies gaps in evaluation suites by injecting errors.
  • Green eval run only confirms current tests passed, not future regressions.
  • Mutation testing and deterministic trace checking complement each other.

Our build wave shipped four scrapers that scraped nothing

Our build wave finished four scrapers in a day. Every test was green. Not one of them scraped anything. This is what that failure actually looked like, and why every gate we had said the opposite.

  • Four scrapers built and shipped without actual scraping functionality
  • Failure due to scaffold stub used instead of real implementation
  • Eleven customer runs recorded as succeeded despite being placeholder runs

Dynamically Naming Servers

  • Created dynamic DNS server with ed25519 keypairs for authentication
  • Server registers, updates hostnames with IP addresses via SQLite DB
  • Uses TCP/UDP handlers, HTTP endpoint for managing hosts/IPs

More from Monday 24 August →