Elo Uncovered Robustness And Best Practices In Language Model Evaluation 2023 11 29
Source
cohere.com
Author
Cohere
Date
Context
An Elo ranking can change even when the underlying judgments of model answers stay the same. Cohere researchers investigated this sensitivity in a 2023 study. Elo updates ratings after each comparison, so the order in which results enter the calculation can affect the final scores.
The researchers reshuffled existing comparison outcomes rather than generating new ones. Their experiments found particular instability when models performed similarly, while averaging across shuffled orders produced more stable rankings. A small Elo lead therefore needs evidence that it survives reasonable changes to the calculation before it supports a confident claim of superiority.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
postCohere releases open-weights translation model North Small Translate
articleThe Data Provenance Initiative A Large Scale Audit Of Dataset Licensing And Attribution In Ai 2023 10 25
