I Benchmarked Jev on Agent Tool-Call Risk. Calibration Held.

On 2026-09-19, researcher Mike published a benchmark of the Jev AI tool, comparing two versions: jev-latest and jev-preview. Both models demonstrated high accuracy (91.7%) and reliable confidence scores, correctly identifying risk postures for agent tool calls. The models achieved 100% accuracy on clear cases (34 examples) and 91.7% accuracy on both ambiguous (14 examples) and adversarial (12 examples) cases.

The key insight is that Jev's confidence score is a reliable indicator of its correctness. Incorrect predictions always come with a confidence lower than 1.000, while correct predictions have a confidence of exactly 1.000. This calibration holds true across both models and multiple runs, making it a crucial factor for any escalation path that may be built on top of the tool's output.

However, the benchmark also reveals some limitations. The ambiguous cases (14 examples) showed only 71.4% accuracy, indicating potential areas for improvement. Additionally, while both models performed well on the adversarial cases, the confidence scores varied slightly between the two versions. The benchmark found one miss in the high-confidence bin (0.9–1.0), with the specific example being a privileged, destructive call with a confidence of 0.97 for Jev-latest and 0.98 for Jev-preview. Nonetheless, this still falls short of the ideal 1.000 confidence level.

In summary, Jev's calibration is stable and reliable, with its high accuracy and precise confidence scores across various risk postures. However, there is room for improvement, particularly in handling ambiguous cases and maintaining consistent high-confidence predictions.