AWS’ Colleen Aubrey: Stop inventing new AI metrics — it’s your business metrics that matter

Colleen Aubrey, senior vice president of Applied AI Solutions at Amazon Web Services (AWS), has stressed that businesses and IT leaders should focus on business metrics, not new AI-specific metrics, when evaluating the impact of artificial intelligence initiatives. This advice comes after a discussion between Aubrey and a reporter, in which Aubrey shared her insights on how to move AI programs out of pilot mode and into successful implementation.

Drawing from her 20+ years of experience at Amazon, Aubrey argues that the key to AI success lies in understanding and improving business metrics. She criticizes the tendency for organizations to create new metrics specifically for measuring AI impact, stating that what truly matters are the standard business metrics. She provides the example of Wells Vehicle Electronics, which cut inventory by 7% in three months while expanding the SKUs they actively monitor from 20 to 2,000 – a result measured not by AI tokens, but by the CFO's preferred metric: working capital.

Aubrey shares the story of AWS Connect, a customer service platform that was highly successful. The team behind Connect focused on a single, critical success factor: outstanding voice quality on the AWS backbone, rather than chasing feature parity with established ERP vendors. This approach paid off, leading to the expansion of the Amazon Connect family, which includes Connect Customer, Connect Talent, Connect Health, and Connect Decisions.

Based on her experience, Aubrey has identified five key lessons that IT leaders should consider when implementing AI projects:

1. Start by identifying the most acute pain points and areas where current tools are weakest, rather than attempting to replicate an entire workflow.

2. Measure success not through abstract AI metrics, but by improving key business metrics.

3. Invest in fixing the underlying issues that are exposed by AI, rather than simply adopting a new tool.

4. Be open to redefining roles and responsibilities to adapt to the new capabilities provided by AI.

5. Prioritize user experience in AI development, ensuring that the technology fits seamlessly into existing workflows.

Aubrey emphasizes that the success of AI initiatives ultimately hinges on the ability to tie them to tangible business outcomes. She advises IT leaders to shift from task-based job descriptions to outcome-focused roles, recognizing that the real value of AI lies in its ability to drive measurable improvements in business performance.