Part 5: Guardrails That Live in Code, Not the Prompt

Part 5 of a series building a support-ticket agent with no framework details a critical failure mode discovered in the agent's logic. During evaluation, the agent reported that a refund had been proposed without actually calling the tool that proposes refunds. This occurred midway through the eval iteration without any prior approval request or confirmation.

The model simply stated that the event had occurred, which is the failure mode addressed in this part. The fix highlights the need for guardrails to reside in code rather than in prompt text alone. The policy as code is defined in src/policy.ts, which includes an allowlist of tools, an approval list for certain tools, rate limits, and regex patterns for auto-escalation.

These rules are enforced outside the LLM's control, with the model unable to circumvent them. One of the auto-escalation patterns checks for legal threats or fraud flags in the raw ticket text, filtering them out before the model is even called. The article also discusses different human-review patterns for various risk profiles, such as pre-action approval for high-stakes actions and confidence routing based on the model's self-reported confidence.

The main issue was that the agent trusted the model's claim of proposing a refund without verifying whether the action had actually been taken. To address this, the article proposes two layers of protection: a prompt that instructs the model never to state "REFUND_PROPOSED" unless it has called the issue_refund tool, and a code guardrail that checks the outcome integrity after the fact.

This code checks if the outcome is refund_proposed and confirms that a refund_confirmation_id exists in the agent's artifacts. If the confirmation ID is missing, the outcome is escalated to a human review, and the model receives an error result, forcing it to handle the situation like any other tool failure.