Vibeleaderboard
← All Intel
Intel / post

GitHub releases ReviewBench, an open benchmark for AI code review

Source
x.com
Date
GitHub@github

How do you know whether an AI code reviewer catches the issues that matter without adding noise? ReviewBench is a new open benchmark shaped by analysis of 103.9M GitHub pull requests, with 219 PRs across 19 languages. Bring your own code review agent, evaluate it, and submit your results ⬇️ https://t.co/YVa464lyfx

Context

ReviewBench is GitHub's attempt to give AI code reviewers a shared test, built around the question its announcement on X puts first: does a reviewer catch the issues that matter without adding noise? That question has two sides, missing real problems and flagging things nobody needed to act on, and a shared lets reviewers be compared on both by running them all on the same pull requests. GitHub says this one is open, contains 219 pull requests across 19 languages, and was shaped by its analysis of 103.9 million pull requests on GitHub. Teams can bring their own code review , evaluate it and submit their results.

The post itself does not say how the 219 pull requests were chosen from that analysis, what counts as an issue that matters, or how a reviewer's comments are scored, so it cannot by itself show how much a ReviewBench score should be trusted. GitHub's blog post sets out the construction, scoring and submission , and Intel covers it in a separate entry, which is the place to judge the method.

Terms in this piece · Glossary
  • Benchmark — A fixed set of tasks with a stated way of scoring them, used to compare models or setups.
  • Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
  • Process — A running instance of a program, with its own working memory, open files and identity.
More from GitHub
Recommended reads
Comments

Checking sign-in…

Loading comments…