ReviewBench: An open benchmark for AI code review
- Source
- github.blog
- Author
- Michelle Zhou
- Date
GitHub built ReviewBench to compare AI code reviewers on a common set of pull requests with one scoring method. It follows the language, repository size and pull request size distribution of more than 100 million real GitHub pull requests. Each pull request has a golden set of known findings drawn from multiple sources, scored with a consistent rubric, and GitHub says senior engineers independently validated it. Results break down by severity and category and by precision, recall, F1 and an F-beta score that lets a team weight noise against coverage according to its own preference. GitHub says the benchmark has made its offline evaluation of Copilot code review better at anticipating the direction of production experiments. That is the main use for teams building their own review agents: an offline signal to check whether a change is likely to help before running it on real pull requests. Teams can onboard their own reviewer and submit results.

Context
GitHub published ReviewBench, a for comparing AI code reviewers, meaning tools that read a pull request (a proposed code change) and flag problems before it is merged. GitHub's post says reviewers differ: some report more issues, some produce less noise, and some are stronger on critical problems. A single pass/fail number would hide those tradeoffs.
Each pull request has a golden set, a validated list of known findings. A reviewer is scored on precision (the share of its reports that are valid, so a measure of noise) and recall (the share of known issues it finds). F1 weights the two equally, while an F-beta score lets a team favor one. Results can be broken down by severity and category.
The validity claims are GitHub's: the pull requests follow the language and size distribution of over 100 million real ones, senior engineers validated the benchmark, and offline scores for Copilot code review have become better at anticipating production experiments. The post gives no numbers for that last claim, so treat it as the builder's account.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
postGitHub releases ReviewBench, an open benchmark for AI code reviewGitHub
articleREAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production UsageSmriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra
articleSlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code RepositoriesMingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
Checking sign-in…
Loading comments…