AI adoption isn’t the same as AI usage

Every engineering organization experiences the same pattern when adopting artificial intelligence. While seat activations and token spend rise, the actual software development process remains unchanged. This discrepancy is the true problem, and current methods for measuring AI adoption unintentionally conceal it.

Currently, only usage metrics are tracked, such as token spend, pull request count, and the percentage of code written by AI. These metrics indicate that someone typed a command into a box, a commit was accepted, or an autocomplete feature was utilized. However, none of them provide insight into whether the work has improved. This phenomenon exemplifies Goodhart’s Law, where a target number ceases to reflect reality as it becomes a goal.

The clearest manifestation of this failure isn't specific to AI. Engineers often encounter teams that pass coverage gates using tests that lack meaningful validation. For instance, assert(true).toBe(true) passes, yet no actual testing has occurred. The metric appears satisfied, while the intended goal has been abandoned. However, the dashboard cannot differentiate between these situations.

To ascertain true AI adoption, one must assess the impact of these tools on actual work processes. If removed, would anything significant break? If the answer is "nothing meaningful," then adoption hasn't truly occurred—it's merely a subscription. The three key manifestations of AI adoption are: an individual using AI for personal tasks like writing commit messages or orienting themselves in unfamiliar code; a recurring job being permanently automated, with the team no longer performing it; or a team entirely reworking a process to incorporate AI more effectively. Only the third option tends to persist over time and changes in workload.

Most organizations celebrate the first two, but rarely achieve the third. This is because the third option requires a cultural shift, openly acknowledging that previous methods were suboptimal and accepting the consequences when the new approach encounters challenges. At Webflow, one successful strategy was treating prompts as shared, versioned artifacts rather than personal property.

By maintaining these prompts in the repository with clear ownership and review processes, the organization has fostered enduring AI adoption.