Long Live Context Engineering - with Jeff Huber of Chroma
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
Jeff Huber, CEO of the open-source search database Chroma, defines as deciding what goes into a model's at each generation step, and improving that choice over time. In a Latent Space interview Huber rests the case on Chroma's Context Rot report, which tested 18 language models and found reliability falls as inputs grow, even on intentionally simple tasks. Huber contrasts this with vendor charts showing perfect needle-in-a-haystack retrieval at long lengths, which Huber says does not mean the whole window is usable.
For the selection step, Huber describes a pattern emerging among teams: use search signals such as vector search, full-text search and metadata filtering to cut thousands of candidate chunks to a few hundred, then have a rerank them to a few dozen. Huber also urges teams to keep a small golden set of queries paired with the chunks each should return, so changes can be measured. These are a practitioner's observations, and Chroma sells retrieval infrastructure.
- context engineering — Deciding what an agent sees at each step — the discipline that replaced prompt engineering once context windows got large enough to fill badly.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…