Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
Source
youtube.com
Author
AI Engineer
Date
Context
The useful size of a searchable passage can depend on the question. A small passage may isolate a name, while a larger one preserves a relationship explained across several sentences. This multiscale retrieval proposal addresses the mismatch between choosing passage sizes when indexing documents and learning the actual question later.
The method indexes several passage sizes and searches them all. It converts the results into rankings of whole documents, then combines their ranking positions. This gives different questions access to different amounts of surrounding context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, at the cost of maintaining multiple indexed versions.
Key takeaways · AI-distilled
An oracle experiment, indexing the same datasets at six chunk sizes and picking the best size per query in hindsight, found fixed chunkingSplitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.Full definition → costs 20-40% of recall versus that per-query optimum.
The team's fix indexes the corpus at every chunk size, queries all of them, then returns whole documents (since chunks of different sizes aren't directly comparable) and merges rankings via reciprocal rank fusion.
On a Seinfeld transcript corpus, a 100-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → chunk answered one question at rank one while every larger window buried it below rank fifty, but the same small chunks failed completely on a question whose answer was spread across a whole scene.
The multiscale-indexing fix costs two to five times the memory of single-size indexing but adds almost no latency, per the team's own accounting.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
chunking — Splitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
videoVoice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval
video78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble
videoWhen Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec
videoHow to Decide What Your AI Agent Should Remember - Wojciech Pluta, Oracle
