Intel / article

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Source: arxiv.org
Author: Pengyu Wang, Benfeng Xu, Shaohan Wang, Xin Zeng, Huarui Wu, Lei Zhang, Licheng Zhang
Date: Published Jul 30, 2026

Context

Searching a growing document collection benefited from ranking likely matches before an began exploring. In a controlled study, researchers expanded the collection across 28 sizes while keeping the questions, relevant evidence and misleading documents fixed. BM25, which ranks text through word matching, overtook a file-browsing agent around 10 million corpus and led at larger shared sizes. That crossover belongs to this experiment's corpus and model setup. Study.

A follow-up comparison helps explain the result. On a 150-question sample at full scale, an agent given BM25 search scored 69.4, against 54.8 for BM25 alone and 36.9 for the file-browsing agent. The evidence favors combining ranked discovery with further reasoning, so the agent spends its effort examining promising documents. Retrieval comparison.

Key takeaways · AI-distilled

There is no single best design — there is a crossover point. An agent exploring a filesystem wins on small corpora, but around 10 million corpus tokens plain BM25 keyword search overtakes it and leads at every larger size, by nearly 20 points at full scale.
The agentic approach loses because its search is sequential: at the largest tier it burns 39 times more query tokens than BM25 and gets less effective as the space it must walk grows.
BM25 also anchors the cheap end of the frontier because it needs no calls to build the index at all, while graph-based RAG hits construction walls before it even reaches deployment scale.
The practical rule the study lands on: use agentic reasoning after ranked discovery, not instead of it. Let a cheap global ranker narrow the field first, then let the agent think about what came back.
The comparison holds constant what most RAG papers vary — same questions, same reader model, same judge, 28 nested corpus tiers spanning a 450-fold range — so the differences are attributable to the retrieval paradigm rather than the benchmark.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

“The results reveal a scale-dependent crossover rather than an unconditional winner.”

“Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale.”

“Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.”

Read the source arxiv.org