I have been Vibecoding Evals (works better than I thought)
I’ve been building AI apps with coding agents for a while. Lately, I’ve been experimenting with evals too. The app in this example mostly worked. That was the problem. The bug I built a small support-triage app for a fictional shipment-tracking company. A customer sends a support ticket, and the app decides what it is about, how urgent it is, and whether a human needs to respond. A real outage…
A developer has been experimenting with AI-powered apps that utilize coding agents. They recently started using evals, or evaluations, to test these apps. The developer built a small support-triage app for a fictional shipment-tracking company. The app was designed to classify incoming support tickets, determine their urgency, and decide whether a human intervention was needed.
However, the developer discovered that the app misclassified a URGENT request about rotating an API key as a security incident, which triggered a human escalation. This was incorrect, as the policy stated that normal key rotation should be a self-service request.
The developer wanted a systematic way to test the app and ensure that it behaved as expected. They installed the DeepEval agent skill, which provided a simple setup for running evaluations. DeepEval generated a checklist, or golden dataset, containing the expected category, priority, and whether a human should be involved for each ticket.
The developer then ran the app with these test cases and observed the failures. The first test case revealed that the app incorrectly classified the ticket as a security incident. This mismatch between the app's output and the expected results was the baseline failure.
Using DeepEval's built-in metrics, the developer could verify two aspects: whether the app's routing fields matched the expected values, and whether the decision complied with the policy. The failure pointed to a broad and aggressive instruction in the triage prompt, which was treating any mention of API keys as a security incident.
The developer modified the prompt to reserve escalation for actual compromise, suspicious access, or an outage. After fixing the prompt, the key-rotation case passed the evaluation, but they also wanted to ensure that they hadn't introduced any new bugs.
The workflow for this process involved building the app, creating test cases, running the app, reviewing the failures, and fixing the prompt. With evals, each prompt change had a systematic checklist, reducing the risk of introducing new bugs. DeepEval is an open-source tool that allows local execution without needing a Confident AI account.
However, some metrics require an API key for a model. The developer shared their full workflow in a video (https://x.com/juampitech/status/2084330588593385637?s=20) and provided a DeepEval Vibe Coder quickstart guide (https://deepeval.com/docs/vibe-coder-quickstart).
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
