Intel / article

smevals - a small eval suite for evaluating models, prompts, and harnesses

Source: simonwillison.net
Author: Simon Willison
Date: Published Jul 31, 2026

Context

smevals separates producing an AI answer from deciding how good that answer is. Simon Willison describes the evaluation tool as a way to run the same tasks against different models, prompts or setups, store what happened, and then grade those records separately. That separation lets developers revise their checks and reassess existing outputs without generating every answer again. Checks can test concrete properties, such as valid XML, or invoke another model to judge an answer. The resulting comparison is therefore defined by both the task and the grading rules, which developers can inspect alongside the recorded outputs.

Key takeaways · AI-distilled

smevals separates running from grading: a run records what one config did on one task, and graders score stored runs afterward, so you can rewrite your checks and regrade without paying to hit the models again.
The unit under test is a config, not a model. A config bundles the model with , parameters, and , so you can hold the task fixed and ask whether the prompt or the harness moved the score.
Checks span from string matching and 'is this valid XML' up to custom checker scripts that call another model to judge the output, so deterministic assertions and live in the same suite.
The intended workflow is agent-first: point your coding agent at `uvx smevals docs` to load the README, tell it to build the suite (a directory of YAML), then run, grade, and either serve results locally or export static HTML.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.

Key quotes

“The most time-consuming part of this project was figuring out the vocabulary for it!”

“An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?”

“I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me.”

Read the source simonwillison.net