Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark

In this test, general-purpose AI tools quoted scientific articles accurately but misjudged which ones mattered and wrote syntheses that experts found accurate but shallow. Researchers at the University of Manchester and the University of Washington compared six tools against a published, hand-built review of the Spanish plume, a weather pattern associated with severe convective storms in Europe. Because that review's 102-article corpus is known in full, every missed article and misjudgment can be counted. The tools were ChatGPT and Gemini Deep Research, Claude's web interface, the coding agents Codex and Claude Code, and Google's NotebookLM.

Search depended mostly on access. In the main search, where login and browser state were not controlled uniformly, no tool found more than 42 of the 102 articles, and the five search tools together found 57. When the first author signed the two coding agents into the university's subscriptions, which also got them past sites that had blocked them, Codex went from 42 to 82 and Claude Code from 15 to 79; some sites refused automated requests even for articles that were free to read, and older articles in one journal survive online only as scanned images. Given PDFs of all 102 articles, Codex, NotebookLM and Claude Code quoted them almost perfectly, apart from seven NotebookLM quotations with altered wording, and all three recognized the 18 articles that discuss the plume substantially. They also classed between 40% and 79% of the 84 articles that mention it only in passing as substantial.

Two experts scored the three written syntheses without knowing which tool wrote which. They found them largely accurate but short on insight; one called an output "just a summary without any insights." All three overstated how far the literature agrees, despite instructions not to claim consensus, and errors that only a specialist would catch got through, including a description of the plume that conflicts with its original definition. The study covers one topic, one run per tool and two reviewers who often disagreed, and the authors note that their error-averse prompt may have suppressed interpretation. What it supports is a division of labor: the tools can help with finding and extracting literature, while judging importance and synthesizing it still need an expert who checks each claim.