ReactBench
reactbench.com- Category
- Developer Tools
- Rank
- No. 1144Tools index
Previous survey · No. 1126 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Builder
- millionco
- GitHub
- 281 stars
- Date
About
ReactBench is an evaluation benchmark for coding agents on realistic React work, mined from merged pull requests across 50+ open-source repositories. Solutions must pass held-out behavioral tests and produce no new React Doctor issues (performance, accessibility, quality), holding agents to a higher bar than test-passing alone.
What it does
A test suite for AI coding agents that grades React code on more than whether it works. Each task drops the agent into a real project at an old commit, with network access cut, and asks it to rebuild a change the project's maintainers actually shipped or to clean up flagged React problems. A separate grading machine then runs tests the agent never saw and a pinned static scanner that compares the result with the starting code. One new rendering, accessibility or code-health finding fails the task, even when every test passes.
Why it's ranked here
The grading design is the argument. Most coding benchmarks stop at passing tests; this one also fails any submission that adds a new scanner finding, and it counts a skipped, missing or renamed test as a failure. Agents run offline, grading happens in a second sandbox with no network, and the reference patch never enters the agent's environment. Tasks carry the original pull request link and commit, and released versions are pinned on a public hub so a score can be tied to an exact task set. The weak spot is size: the published set holds 39 tasks.
What's good
Task writeups are unusually specific. The PortOS task spells out exported helper behaviour, the exact accessible label for a new filter, toggle titles, and that back and forward navigation must restore selections, then states which checks are hidden unit tests and which run in a real browser harness. Graders compare URL values after decoding, so equivalent encodings are not punished. React-aware linters are removed from the agent's image, so the cleanup tasks test whether the model spots problems unaided. Task definitions and test files carry a canary string so training-data contamination can be detected.
Tradeoffs
Running it is heavy. Each task asks for 4 CPUs, 8 GB of memory and 10 GB of disk, agents get up to an hour, and environment builds are allowed 40 to 50 minutes, so the documented eight-way parallel run needs a large machine or a hosted sandbox. Python is pinned to 3.13 only, and the harness depends on an exact framework release plus an older sandbox client kept back to dodge a breaking change. Results lean on one pinned scanner's idea of what counts as a problem. With 39 tasks, a few passes either way moves a score noticeably.
How to use it well
Use it to compare agents or models on frontend work where shipping slow or inaccessible React is a real cost, not as a general coding score. Run the reference solutions five times first, as the maintainers ask, to confirm your sandbox is stable before testing an agent. Pin the hub version that matches the published results you want to compare against. It does not cover backend code, non-React frontends, or interactive agent workflows with a human in the loop.
Technical notes+
Built on the Harbor eval framework: pyproject.toml pins harbor[vercel]==0.23.0 and vercel==0.10.0 (comment: Harbor 0.23 passes project_id to create_sandbox, which vercel 0.11 removed) with requires-python >=3.13,<3.14. dataset.toml publishes reactbench/reactbench version 1.2.0 with 39 [[tasks]] entries, each pinned by sha256 digest. Per-task task.toml (schema_version 1.4) sets [agent] network_mode = "no-network" and timeout_sec = 3600, [verifier] environment_mode = "separate" with its own no-network environment, and a verifier.collect command that runs git diff against /opt/base-sha, scoped to the task's source paths with build and cache directories excluded, writing /tmp/agent.patch as the only artifact. tasks/atomantic-portos-1698/README.md documents hidden Vitest cases plus a Playwright harness mounting the component in a MemoryRouter; the harness entry points (tasks/atomantic-portos-1698/tests/harness/src/main.tsx, tasks/same-recursive-file-drop/tests/harness/src/main.tsx) are thin createRoot mounts. tasks/docusaurus-tabs-11733/task.toml is tagged hard with a 1500 s verifier timeout and requires the workspace package build to compile.
Observed
- Interface
- CLI via the Harbor runner (uv run harbor run), plus a published Harbor Hub dataset
- Runtime requirements
- uv and Docker; Python 3.13 only (pyproject.toml)
- Agent isolation
- Agent runs with no network; verifier runs in a separate no-network environment (task.toml)
- Grading
- Hidden fail-to-pass tests plus a pinned React Doctor scan against the base commit; any new scanner issue fails the task (README.md)
- Task types
- Write React (re-implement a merged pull request) and Fix React (remove target scanner findings)
- Per-task resources
- 4 CPUs, 8 GB memory, 10 GB storage, 1 hour agent timeout (task.toml)
- Contamination guard
- Canary GUID embedded in the manifests, task definitions and test harness files
- License
- Not stated in the README or pyproject.toml
Read from README.md, pyproject.toml, dataset.toml, tasks/atomantic-portos-1698/README.md, tasks/atomantic-portos-1698/instruction.md, tasks/atomantic-portos-1698/task.toml, tasks/atomantic-portos-1698/tests/harness/src/main.tsx, tasks/same-recursive-file-drop/tests/harness/src/main.tsx, tasks/same-recursive-file-drop/environment/app/src/app.tsx, tasks/docusaurus-tabs-11733/instruction.md, tasks/docusaurus-tabs-11733/task.toml.
What it can do
Evaluate coding agents on realistic React tasks
Coding agent and React task set mined from merged pull requests → Benchmark scores/evaluation results
Run agent solutions against held-out behavioral tests
Agent-produced code solution → Test pass/fail results
Scan code for React Doctor issues (performance, accessibility, quality)
React code solution → Report of new React Doctor issues
Grade solutions in clean-room sandboxed containers
Agent solution and task environment → Sandboxed grading verdict
Provide realistic React tasks mined from open-source repositories
50+ open-source repositories with merged pull requests → Write React and Fix React task benchmark set
Enforce a higher evaluation bar than test-passing alone
Agent solution passing behavioral tests → Combined pass/fail requiring no new React Doctor issues
Intel on ReactBench
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.