AI success depends on business outcomes, not usage vanity metrics

In racing, leaders often prioritize burning the most fuel for a win. However, applying that mindset to AI metrics can lead companies astray. Recent AI measurement trends have incentivized companies to track how many computational tokens their employees are using. Companies like Meta even created internal leaderboards ranking employees by token consumption.

The phenomenon became known as "tokenmaxxing." However, this leads to the opposite of desired outcomes. Uber, for instance, spent its entire AI budget in months after prioritizing token usage. The right approach is to focus on business outcomes rather than vanity metrics. Define the strategic path to success and build metrics that support it.

Vanity metrics such as GPU utilization rates or AI features shipped per quarter can distract teams from achieving their goals. Metrics like these appear impressive but fail to measure the actual impact of AI tools. Instead, leaders should tie AI progress directly to outcomes and business goals. For example, if 95% of a company's workforce uses a new tool like GitHub Copilot, but cycle time doesn't improve or code quality suffers, the adoption rate isn't an achievement.

Instead, measure the real impact of AI on outcomes like cycle time, rework rate, and cost per change. Good engineering foundations should come first, and AI tools should be evaluated based on their effect on these outcomes. High-performing teams can set a baseline measure for an objective, then assess whether AI tools improve the situation.

For instance, if a support team reduces first-response time while maintaining quality, it's a clear win. Conversely, if AI doesn't improve outcomes, it's an indicator of wasted resources. The key lesson is to define the finish line first and let metrics drive progress toward that goal.