In the Arena: How LMSys changed LLM Benchmarking Forever
Source
youtube.com
Author
Latent Space
Date
Context
A preferred AI answer may win partly because of its presentation. In this 2024 Chatbot Arena interview, the researchers explain how they studied the contribution of response length and formatting to user votes.
Their style-adjusted ranking adds those characteristics to the statistical model used to estimate model strength. This attempts to separate their predictive contribution from the model’s identity. The adjusted score offers another view of the same preferences, but interpreting the adjustment as a causal effect requires additional assumptions. It helps distinguish questions about overall user preference from questions about performance after accounting for presentation.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoWorld Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
videoExtreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
videoInside Abridge: The AI Listening to 100 Million Doctor Visits — Abridge's Janie Lee & Chai Asawa
videoExo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research
