Vizra Evals and Pest's Evals Plugin: When You Want Which
If you are testing AI agents in Laravel, there are now two packages with "evals" in the description, and the obvious question is whether you need both. Short answer: probably not, and which one depends on a single question. Do you need to know whether your agent is good now , or whether it is worse than it was last month ? This is a genuine comparison rather than a sales pitch. I wrote one of…
Both Vizra Evals and Pest's evals plugin are evaluation tools for AI agents in Laravel. They share some similarities, such as being built on Pest and using similar activation contracts. However, they serve different purposes depending on the specific needs of the user.
Pest's evals plugin scores agent output inside an expectation, providing a set of deterministic and scored expectations. Deterministic expectations include functions like toContain(), toMatch(), and toBeCorrect(), which assess the accuracy of the agent's response. The scored expectations, such as toBeCorrect(), toBeRelevant(), and toSatisfy(), call a model to evaluate the agent's performance based on semantic similarity or plain-English criteria.
Pest's plugin is well-built and does not require additional infrastructure, making it suitable for simple evaluation scenarios.
On the other hand, Vizra Evals offers a different approach by persisting runs, storing every sample, score, judge reasoning, tool call, and token cost in a user's own database. This enables several key features that are not possible with Pest's plugin alone. First, it allows for the creation of a baseline, using the scores from the first passing run as a reference for comparison.
This baseline helps in measuring subsequent runs against the initial reference, rather than relying on arbitrary thresholds. Second, Vizra Evals enables row-level regression detection, where specific rows of data are joined across runs by a hash of their content. This means that if a row drops or significantly changes, the framework will fail the build and identify the specific row that regressed.
This feature is particularly useful for evaluating structured extraction tasks with pinned models and stable prompts. Finally, Vizra Evals provides a dashboard route within the user's application, allowing for score trends per suite, sample assertions and judge reasoning, and the ability to compare two runs side by side. These features offer a more comprehensive evaluation experience, particularly when baselines start to matter and the focus shifts from genuinely exploring prompts to defending the agent's quality over time.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
