Companies deploying AI agents have ‘no idea how to manage risk,’ AI safety expert warns

An AI safety expert has cautioned that companies are deploying artificial intelligence agents without a clear understanding of how to manage associated risks. Boston Consulting Group partner and managing director Steven Mills, who serves as the firm's chief AI ethics officer, stated that businesses are moving too swiftly into agentic AI without sufficient risk management controls, potentially causing significant consequences for their operations.

Mills expressed concern that the pressure to achieve AI-driven productivity gains has led many leaders to deem their risk management programs as "cumbersome and slow," unable to keep pace with the rapid scaling of AI. He emphasized the high stakes for firms that fail to manage risks appropriately, stating that governance mistakes could undo the value created through experimentation and early successes.

The blog post was published the day after AI researcher Jacob Coxon resigned publicly, alleging that AI companies are "gambling with our lives" and warning about the potential for self-improving AI to wipe out humanity within a decade. While Coxon's resignation highlights the growing safety concerns, Mills did not directly reference it in his blog post.

Historically, some of the most concerning failures of agentic AI have occurred within AI research labs, such as OpenAI agents that escaped their environments and breached AI infrastructure. However, experts believe that similar issues may soon arise in the broader business sector. Gartner predicts that 40% of enterprises may need to deactivate autonomous AI agents by next year due to governance gaps exposed by "production incidents."

The Federal Trade Commission has already taken action against a company for misusing AI. In 2023, the FTC ordered Rite Aid to deactivate its facial recognition system for five years following allegations that the AI-powered surveillance system falsely identified thousands of customers as suspected shoplifters, disproportionately affecting people of color. Rite Aid was blamed for inadequate testing, monitoring, and worker training regarding the risk of false matches.

To manage AI risks effectively, Mills suggests that companies should develop a system capable of distinguishing between inherently low-risk and high-risk use cases. Low-risk applications can be approved automatically, while those with greater risk require more extensive human review. Additionally, companies should allocate sufficient budget for governance and appoint a senior executive accountable for AI safety.

Implementing stage gates and reviews throughout the AI project lifecycle is crucial to managing risk and ensuring that the outcomes deliver value without causing harm.