Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects
Source
Emad Alharbi
Author
Emad Alharbi
Date
Terms in this piece · Glossary
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition → LLMs asked to review papers scored everything between 7.0 and 8.1, including rejected submissions, and caught 12.1% of deliberately inserted errors. If you use an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge for quality scoring, expect grade compression and near-blindness to factual defects.
articleSelf- and Other-Labels Induce Bidirectional Bias in LLM JudgesSongeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
articleThe Limits of Automatic Evaluation of Creativity in Large Language ModelsAlessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
articleLarge Language Models Threaten Double-blind ReviewBulambo Mwendelwa Gloire, Prasenjit Mitra