Engineering Rag

Racing several small models for query rewriting, with a raw-message fallback after one second, halved median RAG latency without a single-provider dependency. The pattern applies to any agent that runs retrieval on every turn.

• ElevenLabs found query rewriting, which collapses dialogue history into a self-contained retrieval query, accounted for more than 80% of its RAG latency while it depended on one externally hosted LLM.
• The redesign sends each rewrite to several models at once, including self-hosted Qwen 3-4B and 3-30B-A3B, and keeps the first valid response.
• ElevenLabs reports latency fell at every percentile: median 326ms to 155ms, p75 436ms to 250ms, and p95 629ms to 426ms.
• Racing also smoothed peak-hour slowdowns from external providers: during one provider's outage, conversations continued on the self-hosted models, and ElevenLabs says the extra compute cost was negligible because it already runs that infrastructure.

• RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

• articleIntroducing References Sound Control For Music V2
• articleBuilding Voice Agents That Last Some Lessons Learned From Forward Deployed Engineering
• articleIntroducing Ivr Phone Tree Navigation
• articleFinetunes Music Api

• articleTraining a 4B model to produce 81% faster query plans than Postgrespolyphilz
• clipSemantic routing is RAG, with tools instead of documentsAI Engineer
• articleEnhanced Agentic RagUber Engineering 1

Checking sign-in…

Loading comments…