Isu-bench – coding agents tune an unseen web service with 11 load tests
- Source
- isu-bench.trasta.dev
- Author
- trasta298
- Date
Isu-bench is a new benchmark that hands a coding agent an unseen web service and three servers, then lets it run up to 11 load tests. The score is the result of the last run, so an agent that breaks the service late in the session is scored on the broken version. The task covers work that repository-edit benchmarks leave out: profiling the service, rewriting slow parts, deploying the changes, and keeping the system working through a reboot. The problem is unpublished, which the author uses to limit training contamination. The results cover ten model and harness pairs. Claude Fable 5.2 scored 58.4% and GPT-6.2 Astra scored 41.8%, and each entry reports cost and working time alongside the score. That makes the benchmark useful for teams choosing a model for long operational tasks, where both the result and the bill matter. It is one problem from one author, so the ranking describes this kind of systems tuning task rather than coding ability in general.

Context
isu-bench tests whether a coding can make a slow web service much faster without breaking it, a task with no finish line, unlike benchmarks that stop once a set of tests passes. The agent receives a Japanese specification for ISUMAIL, a newsletter delivery service, a working but slow implementation written in Go with a PostgreSQL database, and root access to the three servers it runs on, each with a different CPU and memory size. It may change anything, from database queries and indexes to caching, the programming language or how work is split across the servers, as long as the service still behaves as the specification describes. It gets 11 load tests and no time limit, and the score of the 11th test is its final score.
Each load test sends 70 seconds of simulated traffic. Publishers schedule newsletter issues that must reach every subscriber before a deadline, while readers open, click, share and subscribe. Points come from emails delivered on time and successful user actions, and any data inconsistency fails the test. Scores are shown as a share of an estimated maximum of 1,060,000 points; the unmodified service scores about 1,744. After the last test all three servers are rebooted, and the run scores zero if the service does not come back with that test's data intact. Agents have no web access, and failed tests count against the 11. The format copies ISUCON, a Japanese web performance tuning contest held since 2011, but because past ISUCON problems and solutions are public and may be in models' training data, ISUMAIL was written for this and is distributed only as machine images, with its source unpublished (README).
In the published v1.1 results, Anthropic's Claude Fable 5.2 running in Claude Code scored highest at 58.4%, after 4 hours 31 minutes of agent working time and an estimated $76.55 in API cost, with no failed tests. Google's Gemini 4 Neon reached 54.7% in about two hours for $18.98 despite four failed tests, followed by Gemini 4 Carbon at 51.9% and Claude Opus 5.5 at 43.2%. OpenAI's GPT-6.2 Astra in Codex scored 41.8% in 35 minutes for $9.70, the shortest time among the top five. GPT-6.2 Sol scored 14.1%, and DeepSeek's V4.1 Flash 0.8%. GPT-6.2 Astra's result is well above the 9.7% recorded for GPT-6 Astra, which ran in an earlier Codex version. That is a different picture from Artificial Analysis's coding agent index, where GPT-6.2 Astra was flat with its predecessor, though one run on one problem cannot settle the question.
The benchmark's author is open about the limits. Each configuration was run once because of cost, and the unmodified service alone varies by about 5% between runs, so the page says small differences are not meaningful. Each model ran in its own provider's agent program, so a score describes a model and together. The 100% mark is an estimate whose derivation is kept private, and costs are calculated from counts at list prices rather than what anyone paid. The useful reading is the size of the gaps: on this one unseen problem, three agents got the service past half of its estimated ceiling, two more reached the low 40s, and the rest stayed below a fifth of it, while the difference between neighbouring ranks may be noise. The rules and the exact prompt are published with the harness.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…

