Intel / article

When Policies Change Probabilities: Modular Decision-Making for LLM Code Review

Source: arxiv.org
Author: Rasvik Kudum, Max Corbett, Hitansh Paliwal, Romaisa Fatima, Thomas Jiralerspong, Sneheel Sarangi
Date: Published Aug 5, 2026

Context

Telling an AI code reviewer that accepting a bad patch is especially costly can change its estimate of whether the patch will fail, even when the evidence stays identical. Across four reviewer interfaces, changing the error-cost policy shifted reported failure probabilities by an average of 13.6 to 16.9 percentage points. The experiment used patches labeled by their results in an archived test . Study.

The researchers separated the tasks: estimate risk without the policy, combine it with an independent monitor score, then apply the decision costs in software. This improved average probability accuracy in their evaluation. Under the 10:1 false-accept policy, however, it accepted no patches and matched simply rejecting all of them. Separating estimation from approval made the decision process clearer without establishing useful acceptance under every risk policy. Results.

Key takeaways · AI-distilled

Across 15,792 responses on 720 patches, telling a reviewer the cost of mistakes changes its reported failure probabilities by 13.6-16.9 points on identical evidence - the policy corrupts the estimate it is supposed to consume.
Under a 10:1 false-accept cost prompt, every one of the four deployed reviewers returns actions WORSE than rejecting all patches outright.
Applying the same high-cost rule to probabilities elicited WITHOUT policy information reduces loss for all four systems - proof the elicitation step itself, not just the decision rule, contributes the excess loss.
Their modular pipeline (policy-blind risk elicitation + independent monitor score + costs applied in code) cuts mean loss by .073 per issue at equal costs while accepting 58-68% of patches.
Practical rule: evaluate risk, outside evidence and action separately - never let the downstream policy leak into the probability prompt.

Terms in this piece · Glossary

agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.

Key quotes

“A probability should depend on evidence; costs should determine the action taken from it.”

“For every reviewer, the actions returned under the high-cost prompt are worse than rejecting all patches.”

“Downstream policy can therefore change the probability it is meant to use, motivating separate evaluation of risk, outside evidence, and action.”

Read the source arxiv.org