Intel / video

The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI

Source: youtube.com
Author: AI Engineer
Date: Published Sep 30, 2026

Context

A coding can pass a benchmark's tests while producing a change that a maintainer would reject. A code-review talk cites research by METR that examined this gap directly. Four maintainers reviewed 296 AI-generated changes from three repositories; roughly half of the changes that passed the automated tests would not have been merged, after the authors adjusted for noise in review decisions using maintainers' ratings of human-written reference patches.

Maintainers sorted failures into categories including code quality, breaking other code and core functionality failure. The study tested agents from 2024 to 2025 and did not let them revise their patches after feedback. Its finding therefore supports evaluating whether a change is acceptable within a repository, alongside test success, without assigning that historical rejection rate to newer agents or iterative review workflows. Study methods and limitations.

Key takeaways · AI-distilled

Voss says reviewer effectiveness collapses past about 400 lines of change while agents now open 10,000-line PRs, so asking humans to review harder does not scale.
Passing tests is not the same as mergeable: METR found about half of -passing PRs would not be merged, and Cognition's FrontierCode shows 88% on SWE-bench Pro versus 29% on real mergeability. Voss expects a mergeability benchmark to quickly become a training signal.
On automated review, Voss explains why multi-pass review and a default stance of suspicion cut false positives, and covers how Cursor and GitHub run review in production.
Automated reviewers can be fooled by that a human would catch, Voss warns, leaving production as the last reviewer. Her recommendation is to rebuild review by building a review rather than abandoning it.
Her examples of taking humans out of the loop include Carlini's agent-built C compiler and Bun's million-line Zig-to-Rust port, which contains 13,044 unsafe blocks.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
prompt injection — An attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Read the source www.youtube.com