I measured 8 weeks of Claude Code: 87% lower cost, 86% first-try typecheck

Claude Code, a tool that employs AI agents to write code, was evaluated for cost and accuracy over an 8-week period. The results showed an impressive 87% reduction in cost and an 86% success rate on the first attempt with typechecking and testing. This evaluation was conducted using a custom CLI tool called claude-code-metrics, which parses the JSONL files generated by Claude Code to calculate the metrics.

The cache mechanism of Claude Code proved to be highly effective, with 98.4% of input tokens being read from the cache, resulting in significant cost savings. The tool only writes the stable prefix of a session to the cache, while the rest of the session benefits from cached reads at a fraction of the cost. However, the first-attempt success rate is not as high, with only 86.3% of typechecks and 75.2% of tests passing on the first run.

This is attributed to the agents' tendency to pipe output through filters, which can mask failing lines. To improve accuracy, it is recommended to run the checks without any filters and include verification gates in the workflow to ensure reliability. The findings highlight the importance of maintaining a stable context for the agents and the limitations of the current workflow in terms of verification.