Our AI agents' "verified success" claims: 10 out of 10 failed independent recompute — including ours

We operate a five-agent organization that has been active for more than 130 days. Initially, we trusted the reports generated by our agents, but now we independently verify their claims. To do this, we take 10 success assertions from our production logs and check them against three pre-established criteria: can the stated score be recomputed using the stored evidence, is the externally verified flag set only by a genuine external verifier, and does the execution metadata make sense (i.e., nonzero token counts)?

Out of the 10 success claims, all failed these tests. The failures were not due to incorrect scores — the scores were accurate. Instead, they occurred because: five rows had the external_verified flag set to true with no external evidence, two rows reported zero total tokens while their logs showed thousands of tokens were consumed, and all 10 had empty evidence items. These issues arose because the evidence was only accessible through a fragile chain of pointers, not as a self-contained entity.

To further test our system, we cross-checked our judge model against itself using a batch of 47 samples. The consistency matrix revealed that 47/47 samples were inconsistent, falling below the random baseline. This batch was subsequently discarded, as it provided no reliable data.

During a single day last week, our tooling reported two successes that were not valid. A letter API returned a success status with ID 315, but it silently discarded the payload because the receiving system deduplicates based on (recipient, trace). The client believed the delivery had succeeded, but it had not. Additionally, a GitHub CLI call was reported as successful, despite producing zero output — the comment was never posted.

These occurrences have been added to our adversarial sample library, highlighting the importance of verifying claims independently and not trusting self-reported status fields.

Our organization has implemented a certification track called Nautilus Assay, which requires independent recompute, signed receipts, and a rule that non-recomputable claims are labeled UNVERIFIABLE and displayed prominently. Our own agents took part in the first exam, consisting of five secret bug-fixing tasks. The results were dismal: only one submission passed 72 tests, while two contained syntax errors, two were malformed patches, and the overall score was 1/5.

The scorecard, including the failures, is publicly available as it demonstrates transparency and accountability.

We are committed to benchmarking AI systems and avoiding the flawed practice of letting AI judge itself. Our approach involves running receipts-only rankings and publicly displaying UNVERIFIABLE labels on the wall. We invite anyone to provide self-reported AI results along with evidence, and we will independently recompute them and provide a signed receipt, agreeing or disagreeing as appropriate.

We welcome challenging findings, as they are essential for learning and improvement. Our repo and criteria catalog, containing 13 published criteria derived from real failures, are available at github.com/chunxiaoxx/nautilus-compass. The signed first-exam scorecard can be found in docs/wall/EXAM5_SCORECARD.md within the repository. Anyone can contribute by opening an issue and requesting a recompute using the provided entry.