A Certification That Changes Every Run Is a Coin Flip With a Signature

The project aimed to create three agents within a platform called HivePlane, serving roles such as a repo guardian, release-notes drafter, and incident commander. These agents were intended to register, certify, and ensure the certification of certified loops using real workloads. However, upon the first day, all three agents failed, not because of the control plane, which was a surprise.

This failure demonstrated the challenge of agent portability, meaning that an agent designed for one repository might not function correctly in another. The real agents, created from the reporter's own repositories, exhibited portability issues, which is a critical finding for agent portability in general. The most significant problem was nondeterminism, where the certification results varied with each run, making the certification unreliable.

The certification results depended on the local machine model, which can misclassify tasks as changelog and fail certification randomly across different runs. The reporter concluded that a certification result that changes from run to run is not a true certification, but rather a random outcome. The decision was made to focus the field test on measuring either the agents or the control plane.

The reporter chose to measure deterministic real agents wired through thin shims, ensuring the same seed always produced the same category of run. The result was that certification became repeatable, with production certifications showing the agents working reliably with the control plane. The field test uncovered two important findings only possible in a live stack, such as the egress allowlist bit and under-instrumentation issues.

These findings highlight the importance of writing evidence before the boundary, not after the verdict, and ensuring that every scenario that appears to hang is actually a scenario whose evidence hasn't been written yet.