Urgent.News

What's breaking now, across hundreds of outlets.

Tech

I spent a day breaking my own eval tool's scoring rule, and it was the best use of the day

This is the developer-facing version of a post I published on AI Stack Lab: Your Eval Suite Is Theater Until a Non-Zero Exit Code Stops the Merge . Same measurements, different angle — this one is about the scoring rule itself. I spent a day breaking my own eval tool's scoring rule, and it was the best use of the day Last week I shipped a $1 CLI called AIEval that does exactly one thing: read a…

This reporter spent a day testing their own evaluation tool's scoring rule, and found it to be a valuable use of time. The evaluation tool, called AIEval, is a simple command-line interface that compares recorded model answers to expected strings and exits with a non-zero code if accuracy falls below a certain threshold. The reporter built a set of tests with a 0.80 accuracy threshold and fed it answers that should not pass.

To their surprise, three out of four verdicts came from substring behavior, not the model being right or wrong. This highlights the trap of measuring accuracy based on short tokens, as the accuracy number can be partially a coincidence. To avoid this, the reporter suggests using expected strings that a wrong answer cannot accidentally contain.

The reporter also discovered that a missing response is considered a failure, not a skip, and that max_avg_latency_ms is inert and not enforced by the harness. The evaluation process can be simplified to a single non-zero exit code in a CI step, making it easier to detect model regressions.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in Tech

I Built a Python Tool to Analyze Googlebot Server Logs

How to parse access logs, identify crawler requests, detect HTTP errors, and build a simple crawling diagnostics report with Python.

  • Python tool analyzes Googlebot server logs
  • Script uses regular expressions to extract fields
  • Results exported to CSV format

PDF/A Compliance Isn't Just Another Export Button

There's a difference between generating a PDF and generating a PDF intended for long-term preservation. That's essentially where PDF/A enters the picture.

  • PDF/A compliance goes beyond simple export button
  • Standards address fonts, color, metadata for preservation
  • Validation crucial, tools like veraPDF aid conformance check

We sped up our own wakes, measured that most of them changed nothing, and slowed the idle ones back down

Mishe is a set of AI agents that run on one host and maintain the system they run on. One of the things they watch is themselves: how often a wake runs, and what fraction of those wakes actually…

  • Mishe AI agents altered their scheduling algorithm on 21 November
  • Wake rate increased by 1.9 times, change rate dropped to 20%
  • Idle roles returned to 2-hour pacing after change, 30-minute pacing otherwise

Inside Limn Engine: How a 360-Level Raycast FPS Runs in a Single HTML File

Inside Limn Engine: How a 360-Level Raycast FPS Runs in a Single HTML File When I set out to build NULLPOINT , a retro-style browser FPS with 12 seasons and 360 procedurally generated levels, I had…

  • Limn Engine powers NULLPOINT FPS with single HTML file
  • Fake offscreen canvas caches static game elements
  • NULLPOINT customizes raycasting engine from scratch

Why COBOL's COMP-3 still knocks down integrations with Pix and ISO 20022 (and an open-source tool)

Who works with backend engineering in banks, acquirers or fintechs in Brazil is well aware of a curious contrast: at the client end, the system settles transactions in seconds via Pix and exchanges…

  • COMP-3 (Packed Decimal) format in COBOL hinders integration with Pix and ISO 20022.
  • Iron-Silicon Bridge open-source tool helps understand and debug COMP-3 format.
  • Tool runs in browser, no backend needed, accessible at specified URLs.

More from Sunday 22 November →