Why my underwriting agent can't approve risky loans, no matter how confident it is

A bank teller can be friendly and sure of herself, but that doesn't grant her the power to open the vault. The vault is locked with a time lock, which releases its contents on a predetermined schedule, regardless of who approaches or what they say. This analogy sets the stage for the post's central message: while AI agents can be helpful and fast, they cannot override certain rules, even if they are confidently wrong.

The focus of the post is on underwriting, which involves determining whether to lend money and under what terms. The author created an AI agent to learn about this process, which proved useful for understanding a common challenge in AI model development: how to prevent models from making dangerous decisions.

Underwriting assesses whether to provide loans and what terms to apply. The author's AI agent processes a structured small business loan application, which includes factors such as annual revenue, the requested loan amount, credit score, and debt service coverage ratio. It then returns one of three outcomes: approve, deny, or escalate.

Escalating means forwarding the case to a human underwriter rather than making a decision. This method is known as human-in-the-loop (HITL), where the system handles routine tasks and directs complex or risky cases to human experts.

The agent is built using LangGraph, a Python library for constructing agents as graphs. Each node represents a step in the process, and edges determine the sequence in which these steps occur. The agent's workflow involves running the financial and policy subagents concurrently. The policy subagent utilizes retrieval-augmented generation (RAG), which involves searching the lender's policy documents for relevant clauses and reasoning with them.

A self-critic evaluates the results from both subagents and may request a reevaluation if necessary. To ensure the reliability of this design, the author implemented a hash-chained audit log, which records a sequential fingerprint of each decision. This setup enables the immediate detection of any alterations to previous decisions.

The human reviewer's role comes into play at the decision node, which is positioned after every model has provided its input. This node serves as the final safeguard before a final decision is made. While the agent produces confidence scores, these scores do not function as a safety mechanism due to two primary reasons. Firstly, models can confidently provide incorrect information.

Confidence merely indicates the model's level of certainty, not the accuracy of its assessment. Secondly, prompt injection poses a significant risk, as malicious actors can insert misleading text within the application to manipulate the model's instructions. This vulnerability underscores the necessity of an additional safety check, known as the "ceiling," to prevent the application text from reaching the decision-making process.