Predictive Human Preference: From Model Ranking to Model Routing
Source
Chip Huyen
Author
Chip Huyen
Date
Key takeaways · AI-distilled
Chatbot Arena's scores are not Elo, despite the name. Since December 2023 it fits a Bradley-Terry model, a maximum-likelihood estimate over match outcomes, then rescales the result so the numbers look Elo-like.
A leaderboard is really a predictor, so score it like one. On the July 2023 arena data, always backing the higher-ranked model calls 74.1% of non-tie matches correctly, and that is the number any per-prompt router has to beat.
Even at peak dominance GPT-4 won only 85.1% of its non-tie matches. Roughly one prompt in seven was answered better by a weaker model, and that gap is the entire business case for routing.
Preference routing is a binary classifier with one extra input: feed it (prompt, model_a, model_b) instead of just the model pair. The prototype encoded prompts with DistilBERT and trained on about 21K non-tie comparisons.
Arena data is noisier than it looks. Teams running these votes in production report most users never read both answers and click at random; it still works because that noise is roughly unbiased and cancels out.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
If you're deciding which LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → to call for each prompt, this lays out how to predict which model a user will prefer per-query and route accordingly — potentially improving response quality while cutting cost and latency versus picking one model for everything. It also frames preference prediction as an interpretability tool for mapping model strengths and weaknesses.
Key quotes
“I found that for all non-tie matches in my test set, the model with the higher Bradley-Terry score is preferred 74.1% of the time.”
“This means that even though GPT-4 is the best model overall, there are prompts for which other models can outperform GPT-4.”
“I found that for all non-tie matches, my preference predictor can predict the match outcome accurately 75% of the time if not using prompts, and 76.2% of the time if using prompts.”
“My primitive experiment suggests that predictive human preference is feasible using a surprisingly small amount of data.”
“With more and more models being developed, each with different capabilities and a cost structure, model routing has clear economic values.”
videoPreferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOceanAI Engineer
articleWhich Prompts Make The Difference Data Prioritization For Efficient Human Llm Evaluation 2023 10 22Cohere
articleTraining language models to follow instructions with human feedbackLong Ouyang et al.