Intel / article

WTF are evals? How to test agents and the software they use

Source: vercel.com
Author: George Fahmy
Date: Published Nov 12, 2026

Why it matters

Agents can complete a request that looks successful in logs but is wrong (hot latte instead of iced). Repeated, graded evals show how often a fix works, which one manual test cannot.

Key takeaways · AI-distilled

Vercel sorts tests by how many calls a check needs: deterministic tests (0), one-turn (1) and end-to-end evals (2+). That count tracks build effort, run time and cost, so the post says to catch each problem at the lowest tier that can catch it.
Pass@k and pass^k answer different questions. In Vercel's illustrative coffee-order run, a 60% success rate gives about 99% pass@5 but only about 8% pass^5, meaning a weekday user would get at least one wrong order in roughly 92% of weeks.
The post ties the lead metric to what a failure costs: pass^k when a miss cannot be retried (a paid coffee order), pass@k when a coding agent can use test feedback to try again, and plain success rate for a draft email the user reviews before sending.
Each eval task gets an instruction, a fixed starting environment, a verifier that scores every constraint separately, and optionally a known-good solution. Per-constraint scores show what broke, such as an order that was the right drink and on time but over budget.
Reading traces exposed a misleading verifier: Codex passed the impossible-request task almost every time, but 9 of 10 passes were the agent giving up without searching. Vercel tightened the verifier to also check a command log the agent cannot write to.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Read the source vercel.com