Part 3: Build the Eval Set Before the Agent Exists
Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet. That's exactly the point. If you write…
Part 3 emphasizes the importance of constructing the evaluation set before the agent is built. Many people mistakenly build the evaluation set after the agent is created, which can lead to unconscious bias and measuring against an already functioning agent. By building the evaluation set first, you establish a real target for measurement rather than tuning the evaluation to match your own demo.
The evaluation set consists of 21 cases divided into three buckets: easy, hard, and edge. Each case checks three things: outcome (did it land on the correct resolution, refund, or escalation), trajectory (did it call the right tools in the right order), and policy checks (case-specific assertions). The edge bucket tests the guardrail by intercepting legal threats, fraud flags, and duplicate ticket patterns before any model call is made.
A real harness run shows that 12 out of 21 cases passed, with one failure in the hard bucket pointing to a significant bug in the agent's behavior. This step provides a fixed target to run against, allowing failures to be identified as clear diffs rather than vague issues.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
