Anthropic says Claude writes and reviews 90% of its team’s code. The catch is how it checks the work.
Anthropic says Claude writes and reviews roughly 90% of the code produced by the team building Claude Code, and the The post Anthropic says Claude writes and reviews 90% of its team’s code. The catch is how it checks the work. appeared first on The New Stack .
Anthropic claims that Claude – the company's AI system – writes and reviews approximately 90% of the code produced by its development team, without requiring engineers to manually inspect every change. In a technical post, Anthropic's staff member Addy Osmani explained the verification system that supports numerous agents simultaneously.
Automated checks provide evidence for engineers to assess the outputs generated by these agents. This approach has drawn attention from developers, primarily due to concerns about the cost of model-driven verification and whether cheaper deterministic tooling could alleviate some of the burden.
To control inference costs, Anthropic runs Claude's changes inside a live application, using browser automation for web apps, simulator-connected systems for mobile builds, and local services for backend work. Verification skills are stored in repository-specific files under .claude/skills/, allowing agents to run and test their own changes before committing them. Osmani also recommended property-based testing and differential testing to uncover edge cases and unanticipated behavior changes.
The costs of verification vary depending on the change, and Anthropic keeps them down by running cheap, deterministic checks before invoking the model. Deterministic checks like building, linting, unit tests, and property tests are performed first, as they are routine software operations that cost little to rerun. Only when these checks fail do model reviews come into play, evaluating if the change takes the correct approach or misses a test case.
Concerns have been raised about Claude potentially manipulating tests to pass, but Anthropic has established rules to prevent this. If a change can modify the check that evaluates it, the check is deemed broken. Claude is restricted from skipping flaky tests, inserting sleep commands, or rerunning tests until they pass. When an agent cannot identify the root cause of a failure, it deletes the test and documents the reason. Each commit's verification results, including cases where no verification occurs, are logged.
Since implementing the verification system, the Claude Code team has run between three and ten reliability checks per change, typically only three to ten per change. For example, when modifying session storage, Claude checks whether older transcripts still load with the –resume flag. Code Review gates require developers to verify changes through /code-review or /verify, with teams able to monitor compliance before enforcing checks.
Engineers can bypass a gate if they provide a valid explanation. Between June and September, merged pull requests to Claude Code's codebase more than doubled, while the percentage of changes requiring reverts decreased by more than 40%. However, the company did not specify whether these improvements were solely due to their verification loops or provided separate figures for production defects and regressions.
To evaluate its own reviewers, Anthropic grades them against historical incidents and maintains an answer key for each case. A reviewer earns credit when its finding accurately describes the bug in question. Changes that remained stable in previous versions serve as controls to detect false positives. A fixed test set, isolated from examples used to train the reviewers, is kept to ensure accurate evaluation.
Anthropic measures recall at various stages of the review pipeline, although it does not disclose false-positive or false-negative rates from Code Review evaluations. Ultimately, Anthropic states that engineers remain responsible for every change that ships, regardless of who typed it.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.