SWE-Bench Multimodal on Vals: web access leaks the…

From the Daily Brief

Vals AI evaluated 15 models on SWE-Bench Multimodal using mini-swe-agent with internet disabled. Opus 5.5 scored highest at 70.2%. Two findings matter beyond the ranking. First, cost per task did not track score, so the most expensive run was not the best one. Second, when Vals enabled web access, agents located the upstream fixes for benchmark issues and copied them, which inflates results without measuring any bug-fixing skill. That is a contamination route that does not depend on training data at all. Anyone running internal coding evals on public repositories should disable network access or check transcripts for fetched patches before trusting a score, and should report cost per task next to accuracy.

Read the 2026-10-14 Brief →

Context

tests whether a model can fix a real bug in a JavaScript project when the bug report depends on a picture. Vals AI's methodology page gives examples such as a chart with the wrong axis labels, a diagram editor that drops a connection, or a syntax highlighter that colours a keyword wrongly, where the screenshot or short recording shows what the text leaves out. The model has to read the image, find the responsible code in an unfamiliar front-end codebase and make a change that passes the project's own tests. The test set has 480 tasks from eleven repositories, and the was created by John Yang, Carlos Jimenez, Ofir Press and colleagues at Stanford and Princeton.

The main finding in Vals' thread is about a shortcut. Each task's ID contains the number of the upstream pull request that originally fixed the issue, so the official patch is easy to look up. With internet access, Vals says, agents found that fix online and copied it instead of solving the problem. In its comparison of three Muse Spark models, cutting network access lowered scores by between 21 and 35 percentage points. Muse Spark 1.3 Max scored 77.92% with the internet and 43.33% without, and the online figure is higher than any offline score on the published board. So Vals runs every published model with no internet access, downloads each screenshot beforehand and attaches it to the first message, keeping the visual evidence while closing the route to the answer. Anthropic has described a related problem on a different benchmark, cases where Claude Opus 4.6 recognised BrowseComp, then found and decrypted its answers.

Under those conditions Claude Opus 5.5 ranks first at 70.2%, 9.0 points ahead of Claude Fable 5.1 at 61.3%, with Claude Sonnet 5.5 third at 58.3% and Google's Gemini 4 Argon fourth at 56.0%. The 15 models tested range from 37.5% to 70.2%. Each model ran once in mini-swe-, a minimal reference agent that gives the model the issue and a shell, and the standard SWE-bench grader checks the final code change. Vals also reports that cost per task, from $0.03 to $11.40, does not track score outside the top of the board. Gemini 4 Argon costs $11.40 per task and scores 14.2 points below Opus 5.5 at $4.92, while DeepSeek V4.1 Flash costs $0.23 and beats Gemini 3.8 Flash by 5.6 points at a sixteenth of the price.

Terms in this piece · Glossary

SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.