SciArena's final result: citations beat fluency, o3 won

SciArena was Ai2's public test of how well AI models answer questions about scientific literature, judged by researchers instead of an automated score. For each question, a shared retrieval system adapted from Ai2's Scholar QA found relevant passages from papers, and two randomly chosen models wrote answers from them. The answers were converted to a standard plain-text format with a uniform citation style, so presentation would sway votes less, and model names stayed hidden until after the vote. Researchers picked the better answer, called a tie or marked both as inadequate, and could write down why.

By the demo's retirement on July 15, 2026, 1,693 users had cast 3,919 votes and left 985 written explanations, according to Ai2's final report. In those explanations, citation and reference quality was the most common reason, at 23.4%, ahead of completeness and depth at 19.1% and directly answering the question at 16.3%. These are shares of the reasons people wrote down, not of all votes. Ai2's reading is that fluent prose is not enough: researchers reward answers they can trace and check, and they notice invented or outdated references. OpenAI's o3 finished first on the final Elo leaderboard, ahead of Claude Opus 4.1, Gemini 3 Pro Preview and GPT-5.

Because every model wrote from the same retrieved passages, the ranking compares how models use given evidence, not complete research products with their own search. Ai2 describes vetting only for an earlier internal phase, in which 102 qualified researchers cast more than 13,000 votes, and not for the public voters behind the final figures. The preferences also proved hard to automate. In SciArena-Eval, a benchmark built from the human votes, the best AI judge, again o3, agreed with researchers 65.1% of the time, while similar judging methods exceed 70% on general chat benchmarks such as AlpacaEval and WildChat. Ai2 is keeping the paper, code and dataset available after the shutdown.