Vibeleaderboard
← All Intel
Intel / article

Where AI Use Is Normalized, Author Seniority but Not AI Disclosure Biases Code Review

Source
arxiv.org
Author
Jenna Butler, Emerson Murphy-Hill, Constance Hadley, Alexa Tipton, Nancy Baym, Sankeerti Haniyur, Jina Suh
Date
Why it matters

If you worry that labeling pull requests as AI-assisted will make reviewers rate them lower, this study found no such penalty where AI use is normal. Showing author seniority did bias reviews, so reviewer-facing author metadata deserves scrutiny.

Key takeaways · AI-distilled
  • In a within-subjects experiment at an organization where AI use is normal, 447 software engineers each reviewed the same four code snippets under an AI-use disclosure in the commit message and author-seniority labels.
  • Earlier studies reported that AI disclosure makes reviewers rate identical work as less competent, hitting already-marginalized authors hardest. This study detected no such effect on perceived code effectiveness or author competence in its setting.
  • A seniority label significantly biased both ratings, code effectiveness and author competence, so the authors conclude that author information shown in code review can still skew evaluations.
Recommended reads
Comments

Checking sign-in…

Loading comments…