The Limits of Automatic Evaluation of Creativity in Large Language Models
Source
Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
Author
Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi
Date
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
If you score creative or open-ended output with an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge, expect a systematic bias toward AI-styled text — and near-zero correlation between the usual automatic metrics and actual human judgment.
articleInter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended TextHaoyuan Li, Snigdha Chaturvedi
articleSelf- and Other-Labels Induce Bidirectional Bias in LLM JudgesSongeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung
articleDiagnostic Foundation for Evaluating LLMs' Research Integrity as Co-ScientistsYash Tripathi, Silu Sharma, Sai Sidhanth Manoharan Jayanthi, Shivank Garg, Lin Li