There is no single best RAGRetrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.Full definition → design — there is a crossover point. An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → exploring a filesystem wins on small corpora, but around 10 million corpus tokens plain BM25 keyword search overtakes it and leads at every larger size, by nearly 20 points at full scale.
The agentic approach loses because its search is sequential: at the largest tier it burns 39 times more query tokens than BM25 and gets less effective as the space it must walk grows.
BM25 also anchors the cheap end of the frontier because it needs no LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → calls to build the index at all, while graph-based RAG hits construction walls before it even reaches deployment scale.
The practical rule the study lands on: use agentic reasoning after ranked discovery, not instead of it. Let a cheap global ranker narrow the field first, then let the agent think about what came back.
The comparison holds constant what most RAG papers vary — same questions, same reader model, same judge, 28 nested corpus tiers spanning a 450-fold range — so the differences are attributable to the retrieval paradigm rather than the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →.
Terms in this piece · Glossary
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Why it matters
RAG architecture choice is usually made on vibes; this puts accuracy-versus-cost scaling behind it on comparable footing.
Key quotes
“The results reveal a scale-dependent crossover rather than an unconditional winner.”
“Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale.”
“Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.”
videoThe unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.devAI Engineer
articleRetrieval-Augmented Generation for Scientific Code UnderstandingAaron Nobile, Andreas Adelmann, Mohsen Sadr
articleUniversal Pathologies, Conditional Consequences: A Triple-Robustness Analysis of RAG for Multi-Hop TraceabilityMeftun Akarsu, Burak Ozdemir