From feral to fiduciary: making AI agents serve your interests

Tech leaders face a dilemma as they prepare to deploy AI agents into production. They desire the increased productivity and lower costs AI agents can provide, but they fear the unknown risks stemming from capabilities that differ from their existing software stack. As AI agents become more capable, organizations crave assurance that their oversight can keep pace.

The key to striking the right balance between innovation and risk lies in assessing, measuring, and maintaining trust in these agents. The more trust agents earn, the greater autonomy they can be granted. Think of trust as a line of credit. When you apply for a card, the issuer evaluates your credit history and repayment ability before setting a credit limit.

Once you have the card, payments are subject to real-time authorization and fraud controls. Unusual transactions may trigger additional scrutiny. Similarly, AI agents should be treated with a "trust score" based on their performance during testing. Upon deployment, continuous checks should be conducted, with each consequential action subject to real-time policy enforcement and monitoring for anomalies.

Enhanced agent performance should be seen as evidence to maintain the agent's operating status, not necessarily to grant it more autonomy or scope.

Imagine if a card could make purchases on its own. You'd want rules to keep it within budget and evidence that it spends responsibly. That's akin to an AI agent. Earning trust for such an agent involves a DevOps-style loop: extensive testing before launch, limited shipping based on the trust score, real-time monitoring in production, addressing identified failure points, retesting for trust, and redeploying.

This loop never ends, as agents' behavior can change over time. Continuous testing and monitoring are crucial to spot declining trustworthiness and respond with appropriate fixes.

Even after an agent appears domesticated, it can unexpectedly go feral. This can happen due to environmental changes (drift), malicious actors exploiting vulnerabilities (malicious actors), ungoverned agents lacking visibility and accountability, or interactions between multi-agent systems causing unintended policy violations. Even well-tested systems can fail in production, and with agentic systems, changes to models, prompts, tools, or workflows can alter behavior post-deployment. Continuous monitoring and testing are essential to prevent feral behavior from creeping in.

To move agents from feral to fiduciary – acting in the principal's best interests, accountable, and without the risk of going feral – a governance framework is necessary. This includes giving the agent an identity, testing it, and continually scoring its trustworthiness. The aim is to ensure the agent acts in the principal's interests, meets organizational requirements, and remains within predefined policy boundaries.

By establishing guardrails and treating the agent like a train staying on its tracks, rather than a Mario Kart racer bouncing off walls, trust can be fostered throughout the agentic lifecycle.