The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
Source
youtube.com
Author
AI Engineer
Date
Context
BM25, a method for ranking documents by lexical matches, can become more useful when an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → supplies better queries. This search talk argues that models can include specific entities, dates and identifiers, then issue several queries to gather evidence. The retrieval function can stay the same while the software using it changes.
The argument puts query formulation alongside the retriever as a source of search quality. A complementary multiscale retrieval proposal addresses another variable: how much surrounding text is indexed together. These are separate design choices in getting useful evidence into an agent's context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →.
Key takeaways · AI-distilled
In an 830-question benchmark over roughly 100,000 documents, stuffing answer-bearing documents into context gave high accuracy even for older models, but handing the same model a search tool instead dropped accuracy, showing retrieval, not reasoning, is the bottleneck.
BM25 (Best Match 25) gets its name because it was the 25th scoring formula researchers tried decades ago and it worked best, not because it's a 25th version release.
Bergum argues an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → agent is a more powerful user of BM25 than a human: it already knows entities, dates, and identifiers, so it writes longer, more specific queries and can fire off a dozen in a row, unlike human query logs that stayed short for decades.
Bergum compares a context window to a 1980s floppy disk (about 1.4 megabytes then, roughly 350,000 tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → now) before quality degrades, meaning something still has to curate what goes in even with much larger windows.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
videoVoice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval
video78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble
videoWhen Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec
videoHow to Decide What Your AI Agent Should Remember - Wojciech Pluta, Oracle
videoThe Search Engine for the Agentic Web — Will Bryk, ExaAI Engineer
articleBM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation ParadigmsPengyu Wang, Benfeng Xu, Shaohan Wang, Xin Zeng, Huarui Wu, Lei Zhang, Licheng Zhang
videoWhere RL Will Take Search — Maximilian-David Rumpf, SID.aiAI Engineer