Behind the Scenes of Vibe Code Bench

Context

Vibe Code Bench, from Vals AI, asks a model to turn a written spec into a complete working application, the from-scratch task that non-programmers attempt with tools, rather than resolving a ticket in an existing repository as does. In this behind-the-scenes post, written soon after launch, Vals explains its design decisions, and the most consequential one is how to grade an app whose structure nobody knows in advance.

Most coding benchmarks run unit tests, but those need known files and endpoints, and fixing that structure in the prompt would make the tasks unrealistic. Vals chose realism. Each spec's features are written in plain language as workflows with sub-steps, and an evaluation agent built on BrowserUse clicks through the running app, marking each sub-step pass or fail. A workflow passes when more than 90% of its sub-steps do, a threshold Vals set from the agreement rate it saw between human testers, and the final score is the share of features built to spec. Vals found the automated tester was sometimes more persistent than human testers at finding working functionality. The method is costly, $10 to $20 per app tested and often more than generating the app itself. The benchmark has 100 specs, about 1,000 workflow tests and about 10,000 single-step checks.

The environment shapes the score too. Models work in a modified OpenHands with a local Supabase server for the database and auth, plus Stripe and MailHog services for payments and email, are asked to use a React Vite stack, and get five hours, with a warning at 30 minutes left because models cut off mid-build otherwise submitted nothing and scored zero. Vals tuned its until all frontier models could build a minimal app using each service, so that a lost point reflects the model rather than vague instructions. Among its observations, early mistakes tended to compound into all-or-nothing results, and Qwen and GLM models outscored Gemini 2.5 and Grok mainly because they tested their work before submitting. Those observations describe the models of that time; the post cites Sonnet 4.5 at $6.66 per app and GPT-5 at $1.53. Vals has since released Vibe Code Bench 1-100, the multi-turn follow-on the post proposed, which tests whether models can apply sequential change requests to an app.

Key takeaways · AI-distilled

Vals chose realism over deterministic unit tests: app features are specified in natural language and browser agents test workflows of sub-steps. A workflow passes at over 90% of sub-steps, a threshold set from agreement rates between human testers.
Evaluation is expensive: Vals says testing one app costs $10 to $20, often more than generating it, and argues evaluation methods have fallen behind the complexity of what models now generate.
Vals normalized runs by a five-hour time limit rather than turn count. Models cut off mid-build often scored zero because they never prepared a deployment, so Vals added a warning at 30 minutes remaining.
Qwen and GLM models outperformed Gemini 2.5 and Grok mainly because they set up their environment, implemented simple features and tested before submitting, though Vals notes most of their apps were completely unstyled.
The automated tester was often more persistent than human testers at finding functionality. At release, Vals reported Sonnet 4.5 averaging $6.66 per app against $1.53 for GPT-5, arguing cost and latency reveal more than close accuracy scores.

Terms in this piece · Glossary

vibe coding — Building software by describing what you want to an AI in plain language and steering the result, instead of writing every line yourself.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.