smevals - a small eval suite for evaluating models, prompts, and harnesses
Source
Simon Willison
Author
Simon Willison
Date
Key takeaways · AI-distilled
smevals separates running from grading: a run records what one config did on one task, and graders score stored runs afterward, so you can rewrite your checks and regrade without paying to hit the models again.
The unit under test is a config, not a model. A config bundles the model with system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition →, parameters, and agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition →, so you can hold the task fixed and ask whether the prompt or the harness moved the score.
Checks span from string matching and 'is this valid XML' up to custom checker scripts that call another model to judge the output, so deterministic assertions and LLM-as-judgeUsing one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.Full definition → live in the same suite.
The intended workflow is AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →-first: point your coding agent at `uvx smevals docs` to load the README, tell it to build the suite (a directory of YAML), then run, grade, and either serve results locally or export static HTML.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
A lightweight harness for running the same eval across model and prompt configurations is the missing piece in most ad-hoc evaluation setups.
Key quotes
“The most time-consuming part of this project was figuring out the vocabulary for it!”
“An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?”
“I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me.”
