How the Claude Code Team Verifies Agent-Written Code
- Source
- claude.dev
- Author
- ClaudeDevs
- Date

Context
Claude writes and reviews about 90 percent of the code on Anthropic's Claude Code team, so nobody there can personally check every change. In this post, Addy Osmani describes what takes the place of that reading: checks the runs on its own work, and rules that stop those checks from being fooled (claude.dev). His case is that these verification loops decide how much work you can hand off. If you have to inspect everything yourself, you can run one or two Claudes at a time; with checks the agent can run and read, you can run dozens or hundreds. The one measured result offered comes from internal testing on structured file generation, where a separate grader, a second pass that scores Claude's output against a rubric of criteria, raised task success by up to 10 points over a standard prompting loop, with the biggest gains on the hardest problems. Osmani adds that tests, benchmarks and specs used to be expensive to produce, but an agent can now write them quickly, and once written they run as ordinary software without spending many more tokens.
Osmani names three kinds of test as the ones most worth investing in. End-to-end tests drive a real user flow, such as adding an item to a cart and reaching the confirmation page, so either the feature works or it does not; unit tests run faster but can pass while the feature is broken. Property tests state a rule that should hold for every input, for example that encoding something and decoding it gives back the original, or that an account balance never goes negative, and a library such as Hypothesis or fast-check then generates thousands of inputs looking for one that breaks it. Their value with agents is that those inputs are not shaped by the same blind spots as the agent that wrote the code. Anthropic researchers have used the same technique with Claude to find bugs in popular Python packages. Differential tests suit migrations: the old and new systems receive the same inputs, randomly generated or sampled from production traffic, and any difference in output is a bug or a breaking change, unless it appears on a list of approved changes. Whatever the type, the post warns that flaky tests teach the agent to rerun them instead of fixing the code.
For the team itself, running the change matters more than any test suite. The post advises writing down how to start the app or simulator in a called verify, kept in the repository near the relevant code, which Claude knows to use before committing; Claude can then exercise web changes through the Claude Chrome extension, mobile apps through an iOS or Android , and desktop apps through computer use, and Osmani says this is how nearly all changes at Anthropic are tested. Failures decide where people spend their : Claude tries to fix a failing check, and if it still fails after three rounds, it stops and brings in a human. A reliability checklist of 39 items, each tied to a past incident or PR and to a trigger visible in a diff, lets Claude check only the items a change actually touches, usually 3 to 10, so no one has to remember old outages.
The more distinctive part of the post is about keeping the checks honest. A change must never be able to edit the rule that judges it, so Anthropic's Code Review reads its REVIEW.md instructions from the base branch rather than from the pull request. A review counts only for the commit it examined, since a pass on an outdated commit says nothing about the new one. A new commit gate starts in an observe mode that only logs what it would have blocked, so false alarms surface before anyone is stopped. The reviewers are graded too, by how many merged changes were later reverted, how many a maintainer rejected, and a size-matched set of changes that stayed clean. Another way to grade them is a formal evaluation built from past incidents and reverts, which the team described in its earlier post on code review: the reviewer earns credit only for findings that match a one-sentence description of the real bug. Misses become rules: a pattern drawn from human review comments or reverts must appear in at least two PRs and be detectable from a diff, three agents then try to disprove it, and a person approves it. Claude may not skip a flaky test, add sleeps or retry until it passes; if it cannot find the cause, it deletes the test and writes down why.
The useful lesson is that a verification layer is software that needs the same care as the code it checks. It has to sit out of reach of the changes it judges, be scored against the bugs that actually hurt, and be pruned as it grows. Apart from the grader figure, the post reports Anthropic's practice rather than measured results for the approach as a whole. Osmani's claim is that once those checks are trustworthy, the human job moves from reading every diff to defining what done means and reviewing the evidence, which is what makes running many agents at once workable.
- About 90% of the Claude Code team's code is written and reviewed by Claude. In internal tests on structured file generation, a separate grader checking Claude's work against a rubric raised task success by up to 10 points, with the largest gains on the hardest problems.
- The team favors end-to-end tests because they are hard to fool: the whole feature works or it doesn't, while unit tests can pass and still break it. Flaky tests teach an agent to rerun instead of fix, and slow tests mean fewer runs, so failures are harder to pin down.
- Property tests state rules such as round-trip encoding, invariants, idempotency and equivalence, then use libraries like fast-check or Hypothesis to generate thousands of inputs. The edge cases are not shaped by the same bias as the agent that wrote the code.
- A change must not be able to edit the check that judges it, so Code Review reads REVIEW.md from the base branch. A 39-item reliability checklist fires only items whose trigger matches the diff, usually 3 to 10, and Claude calls a human after three failed fix rounds.
- New commit gates start in observe mode, logging what they would block. Reviewers are graded on reverts and maintainer rejections. A review-comment pattern becomes a candidate rule once it appears in two or more PRs, then three agents try to disprove it before a human approves it.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
- MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Checking sign-in…
Loading comments…





