AI Is Making Code Cheap. Verification Is Becoming the Expensive Part

Anthropic's post on their CI infrastructure sparked discussion about the impact of AI on coding and testing. On the surface, Claude, Anthropic's AI, writes 80% of their code. However, the significant change lies in how this affects the testing process and CI infrastructure. While code generation scaled 10x, CI jobs increased 25x over six months. This rapid growth in code generation outpaced the ability of the system to adapt, leading to a bottleneck that needed to be addressed elsewhere - verification.

The assumption that faster development would naturally speed up the entire lifecycle proved flawed. While development throughput rose, other areas such as pull requests, code paths, test executions, and review loads also increased, leading to additional failures to investigate. This proves that AI doesn't eliminate bottlenecks, but rather relocates them.

The test suite's role in this new scenario became crucial. Instead of running every test on every change, Anthropic implemented a deterministic test impact analysis, selecting relevant tests based on historical performance. This shift was key to preventing unnecessary computation and ensuring that the feedback loop remained efficient.

However, this isn't a traditional test versus AI test scenario. Instead, the optimal solution involves using both, with a deterministic layer handling critical decisions that cannot afford errors.

The test suite, no longer a gate at the end of the process, became a signal guiding the agent's actions. Writing, testing, reading the result, correcting, and testing again formed a more effective and autonomous loop. This change in perspective altered the cost of a slow test suite. Before, it was a mere annoyance; now, it acts as a throttle for an agent's autonomous loop.

QA conversations shifted from "can we automate this test?" to "can your quality system keep pace with development that doesn't sleep?" The responsibility of ensuring quality moved up the stack, involving designing systems to decide what's risky, what must be tested, and which checks should remain deterministic. The goal was to create autonomous development that's trustworthy enough to run unattended.

However, there's a potential failure mode: AI's ability to generate tests quickly doesn't guarantee high-quality tests. Generated tests often cover variations of the same behavior, missing critical failure modes that could lead to significant issues. The metric of generated tests counts but doesn't necessarily reflect the quality or relevance of the tests.

The true question should be what risk those tests cover that wasn't covered before, and whether their assertions are meaningful and coupled appropriately to the implementation.