Keyword retrieval (BM25, Elasticsearch) is far cheaper and faster than vector search and is a formidable baseline alone. The common production pattern is hybrid: a cheap term-based pass fetches candidates, then a costlier embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → step reranks them.
context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → construction is to foundation models what feature engineering was to classical ML. It is also how a model stays current: one trained through last week cannot answer about this week unless this week is in the context.
Document order matters less in RAGRetrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.Full definition → than in search ranking. Models do read the start and end of a context better than the middle, but whether a document made it in at all matters far more than where it landed.
Structured data takes a different path than documents: turn the question into SQL, execute it, then generate an answer from the rows. If more table schemas exist than fit in context, add a step that first predicts which tables the query needs.
Judge a vector index on four numbers, not one: recall, queries per second, build time, and index size. Build time is the forgotten one, and it decides whether you can re-index as fast as your data changes.
Terms in this piece · Glossary
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
A clear, incrementally-built reference architecture for production genAI systems — showing when and why to add RAG, guardrailsThe checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.Full definition →, model gateways, caching, and orchestration — so you can reason about your own stack instead of copying a diagram blindly.
Key quotes
“Context construction for foundation models is equivalent to feature engineering for classical ML models.”
“Combining term-based retrieval and embedding-based retrieval is called hybrid search .”
“You can think of prompt injection as social engineering done on AI instead of humans.”
“Just as you shouldn't give an intern the authority to delete your production database, you shouldn't allow an unreliable AI to initiate bank transfers.”
