When Policies Change Probabilities: Modular Decision-Making for LLM Code Review
Source
Rasvik Kudum, Max Corbett, Hitansh Paliwal, Romaisa Fatima, Thomas Jiralerspong, Sneheel Sarangi
Author
Rasvik Kudum, Max Corbett, Hitansh Paliwal, Romaisa Fatima, Thomas Jiralerspong, Sneheel Sarangi
Published
Why it matters
Tests whether LLM code reviewers separate risk estimation from cost-based action
Finds cost prompts shift failure probabilities by 13-17 percentage points on same evidence
Proposes modular pipeline that elicits risk separately, cutting mean loss by .073 per issue
Key quotes
“A probability should depend on evidence; costs should determine the action taken from it.”
“For every reviewer, the actions returned under the high-cost prompt are worse than rejecting all patches.”
“Downstream policy can therefore change the probability it is meant to use, motivating separate evaluation of risk, outside evidence, and action.”
Transcript
arXiv:2608.02677v1 Announce Type: new Abstract: LLM code reviewers often estimate patch risk and make approval decisions in one prompt. A probability should depend on evidence; costs should determine the action taken from it. We test whether four deployed reviewer interfaces preserve this separation using 15,792 responses on 720 candidate patches, with one that passed and one that failed an archived test harness for each of 360 repository issues. In matched calls with the patch and monitor evidence fixed, replacing an equal-cost policy with a 10:1 false-accept policy changes reported failure probabilities by 13.6 to 16.9 percentage points on average. For every reviewer, the actions returned under the high-cost prompt are worse than rejecting all patches. Applying the same high-cost rule to probabilities elicited under equal costs reduces loss for all four systems, showing that probability elicitation itself contributes to the excess loss. We also evaluate a modular pipeline that elicits risk without policy information, combines an independent monitor score, and applies costs in code. Relative to calibrated reviewer-only scores, the pipeline improves average probability accuracy and, at equal costs, reduces mean loss by .073 per issue while accepting 58 to 68% of patches. At 10:1, it accepts none and matches reject-all. Downstream policy can therefore change the probability it is meant to use, motivating separate evaluation of risk, outside evidence, and action.