How We Used Real Support Tickets to Gate LLM Agent Releases
How a 14-engineer team with no ML specialists built an eval set from 40 real support tickets that now gates every change to their AI agent.
When deploying an LLM agent to handle customer interactions, the most challenging aspect is ensuring its accuracy. Our team had continuous tracing of the support triage agent, allowing us to replay any ticket and observe its reasoning. However, we lacked a method to assess whether a new version of the agent was better or worse than the previous one before its release.
This issue became evident when return requests began landing in the wrong queue in production. Based on this experience, we developed a practical solution to gate every change to the agent using support tickets.
Instead of creating test cases, we started with real tickets. We exported approximately 200 recent support tickets, removing any personal data, and gave them to our support lead, who had been handling them manually for two years. She selected 40 tickets that collectively covered various types of questions. This smaller set was sufficient to detect two critical failures within the first month while being manageable for one person to label in an afternoon.
Each ticket underwent three labeling steps. The correct queue (orders, returns, billing, compatibility, or human) was recorded, along with whether the agent should answer the ticket or be handed off. For tickets that required an answer, we specified the facts the reply must contain and those it must not claim. This labeling process did not involve creating model answers; instead, we focused on the decisions the agent made.
A missed return ticket, for example, resulted from a gap in our labeling criteria. After updating the labels, we added a separate block for misrouted tickets, ensuring they were reviewed by a human regardless of the agent's assessment.
We graded the agent in two ways. Routing and hand-off decisions were checked by code, determining if the agent's output matched the labeled queue. This check was a quick script that ran in seconds. For the content checks—ensuring the reply included required facts and avoided claiming false ones—we utilized a second model call that evaluated the reply against the labels, returning a pass or fail along with a brief reason. We manually reviewed about one-fifth of these graded outputs to maintain quality control.
The entire evaluation process took under two minutes, a significant improvement over the hour it would have taken otherwise. This rapid turnaround allowed us to run the evaluation every time a prompt change, model switch, or new tool was introduced. The team's evaluation channel displayed the routing accuracy on the 40 selected tickets, with the rule that the agent would not ship if this number decreased.
An important shift in our team occurred when the product manager took ownership of the evaluation set, rather than the engineers. Before, specifications were written by me, and engineers interpreted them. Now, the product manager creates the test set and defines pass criteria, and the agent's output is evaluated against these criteria.
This change forced us to decide, in advance, what constituted correct behavior for our customers. It also influenced how we scoped new workflows. Each new workflow we intended to hand to an agent started with its evaluation set, funded before the feature's development. Consequently, we declined several agent ideas due to insufficient time for set creation, and this decision proved beneficial.
In the early stages, the review load increased faster than the drafting time decreased, but we mitigated this by narrowing our scope to three workflows and providing reviewers with a concise checklist. The key takeaway is that while observability is crucial, most teams lack evaluation frameworks. By creating a set of forty real inputs, assigning three labels per input, and running a two-minute script that checks every change, teams can catch potential issues before they impact customers.
Rewritten by urgent.news; may contain errors — check the original.
Source: hackernoon.com