Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework

Part 7 of this series focuses on the real bugs encountered and the decisions made around using a framework for a support-ticket agent built without any framework. The first iteration log mentioned that exact trajectory matching was the wrong check. The initial evaluation found 12/21 cases passed, with most failures resulting from the agent sending a confirmation email when only a lookup was expected.

The fix was to switch the harness from exact-array equality to ordered-subsequence matching, ensuring that all expected tools appear in order while allowing for additional steps in between. This change prevented false failures due to benign non-determinism.

Next, it was discovered that no retry/backoff mechanism in place led to a transient error crashing the entire run. A 503 error caused by model overload during a specific case brought down the entire evaluation harness. To resolve this, exponential backoff was implemented for 429 and 503 errors, along with inter-case pacing to stay within free-tier rate limits.

Another issue arose when a `-latest` model alias silently changed to a stricter quota after resolving to a newer model. This issue was fixed by pinning an explicit model version instead of using an alias, after checking the provider's live usage dashboard for actual quota. The provider's `latest` aliases prioritize capability over quota stability, with the resolved model changing over time without code modifications.

During testing, an artifact resembling a real bug was observed. When piping multiple answers into the interactive CLI via `printf`, the `npm run agent` command would intermittently hang after the first prompt. The root cause was a Node.js `readline/promises` quirk with quickly-closing piped stdin. The issue was confirmed as a testing artifact, not a real bug, by replaying the same input through an actual pseudo-terminal, which worked every time.

The testing artifact was fixed by reusing a single shared `readline.Interface` instead of opening/closing one per prompt. This change maintained correctness regardless of the original symptom.

The most significant finding was when the model claimed a refund was proposed without ever proposing one. This issue was covered in detail in Part 5 and involved two eval cases. The fix consisted of two layers: an explicit prompt line and a code-level check to verify that a confirmation_id exists in state before trusting the model's claim.

The honest note attached to the fifth finding in the actual log mentioned that the issue was verified with a single targeted re-run to conserve free-tier API quota. This specific gap was left visible in the repository on purpose, emphasizing that claiming a full pass without running it would be a mistake similar to the one made in finding #5.

Ultimately, the decision was made to use a single agent rather than a multi-agent approach. Multi-agent systems, involving a coordinator dispatching to specialist workers, come with a significant cost increase (3-8x) in terms of tokens and latency per run. Workers have access to fewer tools, which sharpens tool selection but results in higher costs per run.

Three signals should be considered before adopting a multi-agent system: the task genuinely needing specialized roles, parallel execution of subtasks for a real speed advantage, and volume multiplied by accuracy improvement outweighing the cost multiplier. At low volumes, the single-agent approach generally yields better results, even if the accuracy edge from splitting the task is present.

As volume increases, the cost savings from the accuracy edge can justify paying the multiplier, as the fixed multiplier doesn't scale as effectively with volume. In this particular case, the repo's agent remains single, focusing on handling five tools without exceeding the point where an agent starts confusing tool names, and the ticket volume targeted by the reference build does not justify the additional cost of splitting into workers.

When it comes to selecting a framework, the raw SDK used in this series offers the benefit of simplicity, minimal abstraction, and maximum control, but it is provider-locked. LangGraph provides more complex state management and wider enterprise adoption but requires more setup cost and has the strongest observability ecosystem. Mastra is suited for TypeScript-native teams, while the Vercel AI SDK tightly couples the streaming UI to a Next.js frontend, making it less suitable for long-running, durable workflows.

CrewAI simplifies the standing up of role-based multi-agent crews, while Google ADK is better for teams committed to GCP and Gemini. LlamaIndex is ideal for RAG-heavy use cases with large document corpora but is less general-purpose outside retrieval. The Google ADK and LlamaIndex are less model-agnostic than LangGraph.