How well do agents use test/verification techniques?

The report investigates the effectiveness of various testing techniques and libraries when software agents are given instructions to implement a specific coding task. The findings indicate that these techniques do not significantly improve the correctness of software implementations, and agents struggle to utilize them effectively.

The study compares a range of techniques, including test-driven development (TDD), property-based testing, QuickCheck, and others, as well as different testing skills. The results show that default instructions without additional guidance yield the best results, with slight improvements observed in fuzzing and property-based testing under high effort levels.

However, the majority of agents fail to apply these techniques in an effective manner, often either writing tests within a different framework or superficially using them without fully benefiting from their value. Additionally, skills recommended by the AI labs underperformed compared to default instructions. The researchers suggest that agents have limited understanding of testing principles and may not be well-suited to learn testing techniques through reinforcement learning environments, despite software quality issues being a concern for widespread adoption of coding agents.