My board never scored an outage as a regression. My evidence couldn't prove it.
I put a preprint on arXiv this week. It is about forging evidence bundles that my own verifier calls clean. I mutated the sites where the verifier is supposed to refuse, counted how many mutations survived, and reported the number. The cheapest forgery that survived was four bytes. Two people commented. Within twenty four hours they had pressed on the same weak point from opposite directions, and…
This week, I released a preprint on arXiv discussing how to create fabricated evidence bundles that my verifier classifies as clean. By altering the points where the verifier should reject, I observed how many modifications could be tolerated before appearing within the score. The most resilient forgery survived after mutating four bytes.
Upon publication, two individuals commented almost instantly, each addressing the same weak point from different perspectives. Despite the apparent discrepancy, they were both referring to the same issue. Nguyen Thanh Vinh pointed out that the score is limited by the existing refusals, and an unrecorded comparison further elucidated this limitation. My methodology of counting surviving sites indeed revealed a discrepancy that had gone unnoticed.
Analyzing SPEC.md, I discovered that there are 39 normative obligations, 37 that must be adhered to, and 2 that must not be violated. However, no artifact exists to map these obligations to specific sites. This lack of recorded data renders the measurement unquantifiable, not merely small. It is the current state of affairs, and I am currently working on addressing this gap.
Simultaneously, Sarvex Jatasra published a correction to an earlier article on pricing agent work, acknowledging Vinh's discovery. Both commentaries on this matter emphasize the importance of including the excluded data in the calculations. My drift board, which continuously tests 17 models against a fixed 35-task suite, encountered a setback when Google's endpoint began rate-limiting, causing one model to score 0.0286 on a single task out of thirty-five. This failure to measure the absence of performance can lead to misleading regression reports.
My drift board incorporates a reliability floor of 0.5, excluding low-reliability points from the accuracy line while still accounting for them in the Reliability metric. This distinction accurately reflects the true performance trend. On July 21, 2026, I introduced a system to distinguish between wrong and absent data, which was consistent with my drift board's practices.
In response to Sarvex's recommendation, I considered tagging every invocation with whether it resulted in an accepted artifact and publishing the runs per accepted change alongside the cost per completed task. Surprisingly, I had already implemented a similar system six weeks prior.
To maintain credibility, I decided to publish a VAC bundle containing hash-pinned artifacts that can be independently verified by an offline verifier. However, this bundle did not include information about the excluded runs, leading to refusal by the verifier. Two out of seventeen models currently have disqualified latest runs, indicating that the evidence presented is incomplete and potentially misleading.
In conclusion, the drift board's adherence to the principle of inclusivity in data measurement is crucial. The implementation of a pinned reliability floor and the inclusion of disqualified runs in the published evidence will help prevent future misinterpretations of model performance and ensure a more accurate representation of the system's behavior.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
