Urgent.News

What's breaking now, across hundreds of outlets.

AI

How to Measure Engineering Productivity With AI Coding Tools

Lines of code, prompts and accepted suggestions measure activity, not progress. Here is what the research says about AI coding tools and a four-layer scorecard to use instead. To measure engineering productivity with AI coding tools, track delivery speed, stability and product outcomes, not lines of code, prompts or accepted suggestions. Those are activity signals, and AI inflates them for free.…

Traditional metrics like lines of code, prompts, and accepted suggestions are poor measures of engineering productivity when using AI coding tools. These metrics are merely activity signals that increase with AI assistance, even when the product output remains stagnant. To accurately measure productivity, the focus should be on delivery speed, system stability, and product outcomes. A four-layer scorecard can be employed: adoption, trust, delivery, and product impact.

Research shows that experienced developers can be 19% slower with AI assistance, despite believing they are 20% faster. High-AI teams often see an increase in merged pull requests, but also a rise in review time and PR size, with no corresponding gains in delivery. Google's guidance emphasizes adoption and trust metrics over productivity metrics. DORA's research indicates that while AI adoption can boost throughput, it often leads to reduced stability.

Lines of code have become a misleading proxy for engineering productivity with AI tools. Cloned code has been increasing while refactored code has decreased, indicating a focus on quantity rather than quality. Larger changes slow down the team, and higher AI adoption is associated with a 1.5% drop in delivery throughput and a 7.2% drop in stability. Acceptance rates and accepted lines of code do not necessarily correlate with productivity if delivery metrics deteriorate.

Instead of solely relying on traditional metrics, a four-layer scorecard can be used to measure AI-assisted engineering productivity. The scorecard includes adoption, trust, delivery, and product impact. Adoption can be measured through weekly active users and the share of changes assisted by AI. Trust can be evaluated using acceptance rates, surveys on trust, and the frequency of AI changes being reworked in reviews.

Delivery should be assessed using DORA's metrics such as change lead time, deployment frequency, deployment recovery time, change fail rate, and deployment rework rate. Finally, product impact can be gauged through target user behavior, adoption of shipped features, retention, and support load. By tracking these metrics over time, organizations can gain a more comprehensive understanding of the true impact of AI coding tools on engineering productivity.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in AI

Germany tests Kamikaze drones and AI targeting technology in Lithuania

The Bundeswehr's latest military trials highlight Germany's growing investment in unmanned weapons, artificial intelligence and preparations to defend NATO's eastern flank.

  • Germany tests kamikaze drones in Lithuania amid NATO-Russia tensions
  • AI targeting technology evaluated alongside drones in Freedom Shield exercise
  • Lithuania chosen as testing ground due to strategic location near Russia

What Product Managers Should Never Delegate to AI: 7 Decisions

AI can gather evidence, draft options and argue the other side. Seven product decisions still need a named human owner. Here is the list, the approval rule and a checklist.

  • AI cannot determine which problem deserves team's attention
  • AI struggles to grasp customers' true meaning in requests
  • AI cannot decide which trade-offs roadmap should accept

More from Sunday 22 November →