Vibeleaderboard
← All Intel
Intel / article

Agents are users, so your software needs evals

Source
vercel.com
Author
George Fahmy
Date

Context

An can complete a request without doing what the user asked, so Vercel argues that anyone whose software agents use needs repeated, graded tests of whole tasks, not logs or a single trial run. In a post on Vercel's blog, George Fahmy of Vercel, the cloud platform for hosting web apps, gives the example of an agent that orders a hot latte when the user asked for an iced one. The order goes through, so the service's logs look fine. And because a model's output varies from run to run, one good manual test cannot show whether a fix worked. The post's answer is an eval: the same task run many times, with each outcome scored.

The post sorts agent tests by how many model calls they need. Deterministic tests need none and check that the software itself behaves, for example that a price quote adds tax and tip correctly. One-turn evals need one and check a single decision, such as which command the agent runs first. End-to-end evals let the agent work through a whole task and then check the final result. Cheap tiers should make up most of a test suite, but whether the user got what they wanted depends on the whole sequence of steps, so only the top tier can measure it. The worked example is a fictional coffee-ordering command-line tool, sfbeans, and the request "a large iced latte with whole milk, ready for pickup by 9:00, under $8 including tip" in San Francisco's Mission district. Three of the four seeded cafes are traps: one has no ice that day, one lists a $7.50 latte that costs $9.27 after tax and tip, one is cheap but cannot have it ready until 9:20, and only one meets all three conditions. Each attempt runs in its own container through Harbor, an open-source framework from the creators of the Terminal-Bench coding benchmark, with the tool's clock frozen at 8:30am so every attempt starts from the same morning. Afterward a verifier script reads the order the agent placed and scores each condition separately (exactly one order, the right drink, on time, on budget), counting a pass only when all of them hold.

Repeating the task exposes a difference that a single success hides. Pass@k is the chance that at least one of k attempts succeeds; pass^k is the chance that all k succeed. In five illustrative attempts by Claude Code, Anthropic's coding agent, running Opus 5.5, three pass, a 60% success rate. Treating attempts as independent, pass@5 comes to about 99% (1 minus 0.4 to the fifth power) and pass^5 to about 8% (0.6 to the fifth power). The first says the agent can get the order right; the second says a user who orders this drink every weekday would get at least one wrong coffee in 92% of weeks. The post picks the metric by what a failure costs. A coffee order cannot be retried, so it leads with pass^k; a coding agent with a test suite can try again, so pass@k fits; an email draft the user reviews before sending only needs a good average success rate. It also warns that five clean trials mean no failures seen yet, not that the agent never fails. These figures are the post's own hypothetical numbers, but outside measurements show the same gap: IBM Research found that a GPT-4.1 agent on the AppWorld benchmark succeeded on 77.4% of runs across five repetitions, yet succeeded on all five for only 53.0% of tasks.

Scores say how often an agent fails; the recorded trace of each attempt says why. In the post's example of an over-budget order, the verifier shows that only the budget check failed, and the trace shows the agent saw the $7.50 menu price and ordered without running the quote command that adds tax and tip. That points to a product change rather than a model score: make the total harder to miss, or add a flag that makes the order command refuse anything over budget. Reading traces while building the demo also exposed problems with the eval itself. The first run scored 0% for every agent setup because the agents did not know sfbeans existed, which was fixed by naming the tool in the request. Codex, OpenAI's coding agent, kept looking for sfbeans as a connected tool instead of running it in the shell. And Codex passed the task where nothing fits the request almost every time, but 9 of its 10 passes came from giving up without searching. The verifier only checked that no order was placed, so the team changed it to confirm the agent had searched first. The post also warns that agents may cheat to pass, for example by writing the order file directly, so the seed data and verifier should be out of the agent's reach.

The post is also a product tutorial: it runs the attempts in parallel on Vercel and routes model calls through Vercel's AI Gateway, and its comparison of four agent setups uses numbers the post labels illustrative. The method stands apart from that tooling. Run each task many times, check the final state against every requirement the user stated, choose the success metric by what a failure costs the user, and read the failed attempts and some of the passes before trusting a score. The post's own impossible-request task shows why the last step matters: a verifier that checked too little counted nine shortcuts as wins.

Key takeaways · AI-distilled
  • Vercel groups agent tests by calls: deterministic tests (none), one-turn evals that check a single decision such as the first command run (one), and end-to-end evals that grade a whole task (several). It advises using the lowest tier that can catch a given problem.
  • Pass@k and pass^k answer different questions. In Vercel's example, a 60% per-trial success rate gives about 99% pass@5 (at least one success) but about 8% pass^5 (all five succeed), so a daily coffee order goes wrong at least once in 92% of weeks.
  • Vercel ties the lead metric to what a failure costs: pass^k when the user cannot retry, pass@k for coding agents that can revise against test feedback, and plain success rate for drafts a person reviews before sending.
  • Each eval task needs an instruction, a fixed starting environment, a verifier that scores each constraint separately, and optionally a known-good solution. The set should include at least one request where the right move is not to act.
  • Reading traces exposed a misleading pass: Codex passed the impossible-request task, but 9 of 10 passes gave up without searching. Vercel tightened the verifier to check a command log the agent cannot write to.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
  • MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
More from George Fahmy
Recommended reads
Comments

Checking sign-in…

Loading comments…