Retrieval is the step in an agent system that picks which pieces of evidence should be sitting in the model’s context window for the current request. The word retrieval is a plain English word meaning to fetch back, and in this setting it covers semantic similarity, lexical search like keyword or BM25 matching, metadata filters, graph traversals, and hybrid combinations of all of these. Its single job is recall and relevance: it tries to bring back the right evidence with enough breadth that the generator can ground its answer. The framing matters because generation cannot recover evidence that retrieval never supplied. If the right paragraph is not selected, no amount of clever prompting, tool calling, or reasoning will invent it without hallucination, so retrieval is the floor of grounded agent quality. The canonical pattern catalog treats retrieval, usually embodied as Retrieval Augmented Generation or RAG, as one of seven production patterns that must be designed deliberately rather than as an isolated component glued on later.[1]
Inside an agent system, retrieval sits between the user request and the generator. Step one is to take the raw query and, when helpful, rewrite or expand it, a step sometimes called query transformation, so that natural phrasing like how do I rotate the staging secret matches the terse vocabulary of the indexed documents. Step two is to convert both query and documents into the representation the index expects, which for semantic search means embedding vectors and for lexical search means token postings, and then scan the index for candidates. Step three is to apply metadata filters such as repository, time window, or access tier so the candidate pool is small and on topic. Step four is to return the top k results, often after a first cheap pass. Practitioners choose semantic search when meaning matters more than exact wording, lexical search when terminology is fixed and rare, metadata when the index is well curated and structured, and hybrid when the corpus is mixed and the cost of a miss is high. They should not choose a single dense vector index when exact identifiers like error codes or API names dominate, and they should not choose lexical-only when users ask conceptual questions that share little vocabulary with the answers.[1]
The most common failure mode is the silent miss, where retrieval returns documents that look plausible, are confidently formatted, and even share keywords, but none of them actually answer the question. The generator then produces a fluent but wrong answer because the evidence was never there to begin with. A second failure is context flooding, where retrieval returns too many chunks in the name of safety, so the model loses the signal in a sea of near duplicates. A third failure is index drift, where the indexed corpus no longer matches the source of truth because documents changed upstream and the embedding job lags behind. The durable lesson is that retrieval is a production system with its own failure modes, and designing it as one of the named patterns alongside evals, caching, and guardrails is the only way to keep it honest at scale.[1]
Retrieval versus the surrounding context window
Retrieval is often confused with the context window itself, but the context window is the finite buffer of tokens the model can attend to on a single call, while retrieval is the process that decides which tokens earn a seat in that buffer. Retrieval without a context window is just a search engine; a context window without retrieval is just whatever happened to be in the prompt. The right mental model is that retrieval selects, the context window holds, and the generator reads. Confusing the two leads to two classic mistakes: stuffing the whole knowledge base into the prompt and calling it retrieval, or building a beautiful index and then never wiring it into the live request path so it never actually runs.
Worked exampleSelecting evidence for a debugging question
A developer asks the agent, Why is the staging environment returning 502 after the last deploy? The corpus is a mix of runbooks, recent incident reports, deploy logs, and architecture diagrams. A pure keyword search on 502 and staging will return a flood of generic 502 runbooks. A pure semantic search will return conceptually related but outdated material about load balancers. The goal is to retrieve the specific recent incident that matches this deploy and this environment.
- 01Rewrite the query into a search-friendly form, such as 502 staging incident report after deploy plus the timestamp of the most recent deploy from a metadata call.
- 02Embed the rewritten query and run a semantic search over incident reports indexed by service, environment, and date, while simultaneously running a lexical search for the exact error code 502 and the service name.
- 03Filter candidates to the staging environment, the last 48 hours, and the specific service, then merge the two ranked lists using a hybrid scheme such as reciprocal rank fusion so a document strong in one signal is not drowned out.
- 04Return the top 5 chunks, including the matching incident summary, the most recent runbook section, and the deploy log excerpt, so the generator has grounded evidence without flooding the context window.
The lesson is that retrieval quality is dominated by query rewriting, index choice, and filtering, not by the embedding model alone, and that hybrid search with metadata constraints is usually the right default when evidence is structured and time-sensitive.