chunking
Splitting documents into passages small enough to embed and retrieve individually — the step that quietly determines whether retrieval works at all.
Chunks that are too large blur many topics into one vector and match everything weakly. Chunks that are too small sever a fact from the context that makes it meaningful. Both failures show up as retrieval that misses obvious answers, and neither is visible in the embedding step.
Splitting on structure — headings, sections, paragraphs — beats splitting on a fixed character count, because it keeps whole ideas together. Overlapping chunks slightly stops a fact from being cut in half at a boundary.
Put the idea to use
Imagine a handbook whose rule appears in one paragraph and whose exception appears in the next. Splitting between them can give an assistant the rule without its qualification. Keep enough surrounding text for a retrieved passage to make sense.
A smaller chunk is more focused, but may lose context. A larger chunk preserves more context, but can include irrelevant material. Test boundaries with the questions readers actually ask.
Background: LangChain: retrieval and generation.