{
  "id": 17833,
  "title": "smevals - a small eval suite for evaluating models, prompts, and harnesses",
  "url": "https://urgent.news/2026/07/31/smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses",
  "topic": "ai",
  "section": "AI",
  "published": "2026-07-31T21:15:23.000Z",
  "source": {
    "name": "Simon Willison",
    "slug": "simon-willison",
    "url": "https://simonwillison.net/2026/Jul/31/smevals/"
  },
  "original_language": "en",
  "account": "smevals is a compact evaluation suite designed for assessing models, prompts, and harnesses. The framework was developed by the author in collaboration with Jesse Vincent's Prime Radiant applied AI research lab. The primary objective is to enable users to run small evaluation suites across various model configurations and assess the outcomes.\n\nThe blog post provides a comprehensive overview of the tool. To begin, an eval - consisting of YAML files - must be created. This eval can then be executed against different models, such as GPT-4, Claude, or Llama. The runs are distinct from grading operations, allowing users to evaluate the results based on predefined checks.\n\nAfter running the eval, users can utilize a localhost web server to view the results or build a static HTML report using the smevals build command. In the author's case, the most challenging aspect was determining the appropriate vocabulary for the eval suite. Ultimately, they settled on a set of challenges, such as generating SVGs or creating haikus, to evaluate the models' capabilities.\n\nEach eval comprises a series of tasks, with each task representing a unique challenge, e.g., \"Generate an SVG of a pelican riding a bicycle.\" When executing the eval, users specify one or more configurations, which define the model to be evaluated and may include additional parameters like system prompts, model parameters, or agent harnesses. A run captures the results of executing a specific configuration on a particular task.\n\nTo analyze the collected runs, a grader is employed. The grader determines the grade by executing a sequence of checks, which can range from simple operations like checking for specific strings in the output or validating XML output, to more complex custom operations involving other models. Simon Willison, the creator of smevals, has been refining the concept for years, and he is excited to expand its capabilities and apply it to his own projects.",
  "summary": "smevals - a small eval suite for evaluating models, prompts, and harnesses I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models. The result is smevals , a new tool for running small eval suites across different model configurations and grading the results. The blog entry…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}