Stop Fine-Tuning to Fix Retrieval Problems — Anant Srivastava
Source
youtube.com
Author
AI Engineer
Date
Context
When an assistant gets changing product facts wrong, the first question is where those facts should live. This architecture talk proposes keeping current, large, citable or access-controlled knowledge in memory the assistant can retrieve. Prompts handle small, stable behavior; model weights are reserved for reflexes that have stopped changing.
The distinction gives teams a diagnostic for choosing an intervention. A problem finding the right current document calls for work on retrieval and access to information. Teaching a model a settled, repeatable behavior is a different job. fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → is therefore one tool within the architecture, rather than the automatic next step whenever answers disappoint.
Key takeaways · AI-distilled
Srivastava's diagnostic: put small, stable behavior in the prompt; put knowledge that is current, large, citable or access-controlled in memory; reserve weights for reflexes that have stopped changing.
His rule is to fine-tune reflexes, not facts. Medical coding is his example of a good fine-tuning target, and he says access control belongs in memory rather than in the model.
He argues every prompt edit, indexed document and training sample is an architecture decision, so teams that never decide where knowledge lives end up with an architecture that was accumulated rather than designed.
He closes with a circulating architecture in which knowledge moves between context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, memory and weights over time and the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → improves by doing its job.
Terms in this piece · Glossary
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.