SciArena's parting result: citations beat fluency, and o3 won
- Source
- x.com
- Date
We built SciArena to test how well AI models handle scientific literature questions, as judged by researchers. It's retiring July 15, and the results are in: ~1,700 users cast ~3,900 votes. Here's what they told us. 🧵
What did researchers value most in a model’s answer? 📚 Citation quality—references that are real, relevant, & checkable (23%) 🔬 Depth (19%) 🎯 Directly answering the question (16%) Fluent prose alone didn't win.
o3 finished on top of the SciArena leaderboard – ahead of Claude Opus 4.1, Gemini 3 Pro Preview, & open-weights models like DeepSeek-R1 – with answers researchers called more detailed + to the point. Learn more in our updated blog: https://t.co/DT9LtfSGih
SciArena also allowed us to collect high-quality, expert-annotated ground truth on evaluation data. This is a unique resource & especially important as AI agents are increasingly used to judge the quality of other AI agents, and such evaluations need to be grounded + verified.
Context
Researchers evaluating scientific answers valued references they could check. In Ai2's SciArena, users compared answers with model names concealed and could explain their votes. Citation quality appeared in 23.4% of those written explanations, more often than any other reported category. That percentage describes comments, rather than an accuracy rate or share of all votes. Final project findings.
OpenAI's o3 led the final preference leaderboard. The result concerns literature answers produced within SciArena's shared retrieval workflow, so it does not rank complete research- products. The more transferable finding is what participants wanted from an answer: relevant, verifiable references accompanying a substantive response to their question. Evaluation setup.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleLitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review PlatformRuotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
postArtificial Analysis launches AA-3D, a benchmark for agent-built 3D scenesArtificialAnlys
postTerminal-Bench-Science leaderboard debuts: GPT-6 Astra and Claude Opus 5.5 leadArtificial Analysis
Checking sign-in…
Loading comments…