Ai Code Review

OpenRouter has published a tutorial for an AI code reviewer that tries to prove each bug by running code before it reports it. AI reviewers that read only the diff, the lines a change adds or removes, have to guess what the rest of the file looks like and whether a suspected bug actually fails. OpenRouter's 233-line Python script instead uploads the full changed files, any test files you name, and the diff into a hosted Linux sandbox through OpenRouter's API, which sends them on to the chosen model's provider. The model rebuilds the repository there, writes short checks and runs them. Each finding comes back marked confirmed, with the command and output that showed the failure, or unconfirmed when a quick script cannot demonstrate the problem, as with a database connection that is no longer closed.

The script also works as a gate. The model proposes fixes as patches, which the script tries on a temporary copy of your files and copies into your working tree only when you pass its apply flag. Those patches are not applied inside the sandbox, so the model's checks ran against the original code, and the guide tells readers to run their own tests after applying a fix. When any finding reaches a chosen severity, high by default, the script exits with an error code. Installed as a Git pre-push hook in OpenRouter's test, that stopped a push containing planted bugs, and the remote branch stayed at its previous commit.

The evidence comes from one small test. OpenRouter planted five bugs in a short Python file whose existing tests still passed, then ran three reviews per model. Every model confirmed the SQL injection, the pagination error, an identity comparison that counted refunded orders as revenue, and a retry helper that silently swallowed its final error, in every review it completed. Anthropic's Claude Sonnet 5.5 found all five planted bugs in each of its three runs, and its patches fixed all five every time, at an average of $0.0447 and 60 seconds per review. Claude Haiku 5.5 was the cheapest at $0.0153, but its patches fixed all five bugs in only one of three runs, and two of them rewrote a function in a way that would break callers passing the data shape it expected. GPT-6 Luna found 12 of the 15 across its three runs, and one DeepSeek V4.1 Flash review spent all eight of its turns resending patches without finishing.

Running code costs more than reading it. Sandbox time is billed at $0.0001 per second from the first shell command until the response completes, so a slow model pays for its thinking time even when its tokens are cheap. OpenRouter puts a diff-only reviewer on Haiku 5.5 at about $0.001 per review and the sandboxed version at 10 to 40 times that. What the extra cost buys is a reproduced failure for each confirmed finding. The guide itself says models can still miss bugs, overstate severity or write broken patches, and the uploaded code goes to the model provider. A five-bug sample shows the method catching planted bugs; it does not show how often it catches bugs in real changes.