Financial Services’ next AI risk is the workflow nobody can explain

As financial services increasingly adopt AI agents that execute actions, a new set of accountability challenges emerges. Firms must not only understand how an AI decision was reached but also trace the entire workflow to ensure proper oversight. This shift from AI answering questions to AI taking action creates a more complex governance challenge, as agents move between systems, interpret information, and decide what happens next.

The lack of visibility into these systems poses significant risks. For example, an error by an AI agent could lead to an account restriction, customer communication, or regulatory report before anyone realizes the mistake. The FCA emphasizes the need for firms to map important business services, identify vulnerabilities, and understand how disruptions could propagate through them.

To address these challenges, firms need controls that operate at the level and speed of the process, with clear ownership and an auditable record of decisions and actions. Traditional audit records must be expanded to capture the entire workflow, including which agent acted, what information was retrieved, which systems and tools were used, the rules applied, and where a human reviewed or overrode an action.

While 'human-in-the-loop' is often used to describe responsible AI, meaningful oversight requires that reviewers have complete visibility into the agent's actions and the ability to intervene if necessary. The goal is to ensure that human judgment is exercised where the consequences are greatest, not just to provide a simple veto point.

By implementing these controls, financial services firms can balance the benefits of agentic AI with the need for proper accountability and oversight.