AI Coding Metrics Have a Missing Middle

AI coding metrics have become increasingly sophisticated, yet there is a significant gap between measuring AI usage and understanding the impact of that usage on software delivery. While we can easily track active AI users, tools, and various AI-assisted activities, we often lack insight into what happens in between these stages. This missing middle is where most of the interesting AI productivity questions reside.

Initially, AI metrics focused on adoption, answering questions about who has access, who is using it, which tools are being used, and the cost associated with AI assistance. However, once most developers in an organization are using an AI coding tool, increasing adoption yields little additional insight into the effectiveness of the investment. The real questions lie in understanding what changes occur in the engineering system because of AI usage.

Software development is a pipeline, with stages such as coding, reviewing, testing, deployment, and maintenance. Accelerating one stage may expose or create bottlenecks elsewhere. For example, if an AI assistant reduces coding time by 30%, but pull requests become larger, review time increases by 25%, and rework increases, the overall productivity gain is unclear. This highlights the importance of considering the entire system's performance, not just individual stages.

AI-generated code is both useful and dangerous. While it may increase the percentage of changes made with AI assistance, this does not necessarily translate to improved productivity. The type, size, review time, rework, quality, delivery, and stability of AI-assisted changes need to be analyzed to understand their true impact. AI contribution should be viewed as an analytical dimension, providing insights into how changes behave compared to non-AI-assisted changes.

The most important metric may be where the work moved. Traditional developer productivity metrics, such as commits, lines of code, and tickets closed, may not accurately reflect the value generated by AI assistance. As AI makes producing artifacts cheaper, the question shifts to: Where did engineering effort move? This shift in measurement aligns with a broader trend in software engineering, where activity metrics become more useful when interpreted alongside flow, quality, delivery, and developer experience.

To effectively measure AI's impact on software engineering, we need an AI engineering funnel, rather than relying on a single productivity metric. This funnel should consider various stages of the AI-assisted development process, allowing us to identify bottlenecks, assess the true value of AI assistance, and optimize the overall engineering system.