I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story

Two weeks ago, the author published "Why Agent Evaluation Is Harder Than Model Evaluation," arguing that with agents, you're not just judging an answer but also the path, tools, and safety boundaries used. The agent evaluation harness, AgentEval Forge, is now public on PyPI, and the author shares the learnings from building it.

The project started as a scoring system but became an integration reality check when the author discovered real agents had a significant impact. To build the harness, the author went deep and defined 20 critical user journeys, creating an architecture with five core components: scenario pack engine, runner, scoring layer, regression engine, and adversarial generator.

The development process involved 118 tasks across 12 milestones, with adapters for five agent surfaces: subprocess, Python import, HTTP, LangGraph, and PydanticAI. The author faced challenges sourcing real agents from GitHub due to heavy frameworks, infrastructure requirements, and missing clear entry points. After searching over 150 repos, the author landed on 19 viable agents: 11 LangGraph and 8 PydanticAI.

The author bucketed agents based on stars to create field-test scenarios. High-star agents were for reliability testing, medium-star agents for mutual-friction testing, and low-star agents for stretch testing. The author built CI integration, security models, and documentation to ensure a comprehensive evaluation framework.

The PRD (Product Design Document) was crucial in keeping the author honest throughout the project. It forced the author to answer uncomfortable questions, define adoption outcomes, and establish an evaluation hierarchy focusing on safety, correctness, and efficiency. The 20 critical user journeys and the structured approach provided by the PRD slowed down the development process, but it ultimately ensured a robust evaluation harness.