Tooling every AI software harness should have
When an agent writes the code, you stop reading every line. You can pretend otherwise for a while, but on any real codebase you end up skimming a 600 line diff and approving it because nothing jumped out. Nothing jumping out is not the same as nothing being wrong. Documentation and a well tended CLAUDE.md only take you so far. The longer an agent runs, the more those instructions get crowded out…
Every AI software harness should include a comprehensive set of tools and checks. First and foremost, the quality thresholds have shifted. While 100% branch coverage was previously considered the standard, the focus is now on enforcing higher quality controls at a lower agent time cost. This means adjusting thresholds accordingly, with options ranging from 100% minimum line and branch coverage to 80%.
Moreover, a tool's value is determined by whether it can be seen and acted on by the AI agent itself. For instance, a linter that fails in CI can only provide feedback after the agent has stopped, while a linter integrated within the agent's workflow allows for immediate fixes before being noticed by the agent.
All the suggested tools should be easily accessible through a single command, and fail in a manner that discourages ignoring them. Testing should be performed with a testing framework set to a minimum line and branch coverage of 100%, ensuring no important branches are left untested. Tools like RSpec and SimpleCov can help achieve this.
Mutation testing is another crucial tool. Instead of relying solely on coverage, which may overlook certain types of defects, mutation testing changes specific lines of code to detect if tests catch the modified behavior. Tools like Stryker and Mutant can be used for this purpose.
Type checking is a quick win that offers significant benefits. By catching type errors statically, agents can receive immediate feedback on erroneous code, rather than having to sift through stack traces. Tools like Sorbet and TypeScript can be employed for this task.
Linters and custom linters are essential for identifying outdated or harmful patterns in code. With agents trained on outdated code, linters become even more critical. Frameworks like Rubocop, ERB Lint, ESLint, and Hadolint can help enforce best practices across various languages.
Security checkers are also necessary to prevent insecure coding practices. By automatically detecting common security vulnerabilities, such as insecure queries or secret leakage, tools like Brakemanscanner, Gitleaks, and Zizmor can keep secure coding practices in check.
Duplication detection is another important aspect of a robust harness. As agents tend to produce repetitive code, detectors like JSCPD and Flay can pinpoint duplicated code, helping maintain clean and efficient codebases.
Lastly, hooks should be utilized to integrate these checks at the earliest stages of the agent's workflow, ensuring the highest quality of generated code. By attaching checks at different stages - editing a file, committing, pushing - and leveraging the fact that early-stage checks are more cost-effective, agents can be forced to adhere to these standards.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
