How well do agents use test/verification techniques?

The article examines the effectiveness of various test and verification techniques when used by AI agents to implement a code. It uses the Zstd implementation evaluation, comparing 26 different testing conditions in Rust. Implementations were tested with GPT-5.6 Sol at medium and high effort levels. The results showed that "Default" (no additional instructions) performed above average.

Among the tested techniques, fuzzing and property-based testing (PBT) conditions performed slightly better than formal methods on average. However, skills recommended by the AI, such as Hegel, ECC Rust test skill, Trail of Bits property test skill, and a custom skill, underperformed. The study found that AI agents generally did not use formal methods, test libraries, or techniques effectively.

They tended to write tests they would normally write but inside a different framework for a different type of test technique or apply the techniques superficially without realizing the full value of the technique. The results remained consistent when tested with IMAP RFC and other random RFCs. In conclusion, AI agents lack understanding and proficiency in using test and verification techniques effectively.