Intel / article

Deepgram's RCS and split LLMs aim to fix voice agent context loss

Source: deepgram.com
Author: Deepgram
Date: Published Nov 4, 2026

Context

Most voice agents chain three models: speech-to-text transcribes the caller, a writes a reply, and text-to-speech reads it aloud. Each handoff passes only words, so a sigh or a hesitant tone never reaches the language model, and the speech model gets no instruction about how the reply should sound. In a blog post, Ed Anuff, chief product officer of the speech AI company Deepgram, presents two changes the company announced at its Deepgram Speak event: an open format for passing that missing between models, and a split of the language model's work between two models.

The format is the Real-Time Context Standard, or RCS, developed with Pipecat, an open-source framework for building voice agents that is now part of Deepgram. Anuff's post describes it as three fields added to the chat completions format that most language-model APIs use. The published specification, labeled version 0.1 and an alpha, works differently: it writes the context inline, in the text the models already pass to each other. A reading of how the caller sounded, with labels such as frustrated and a confidence for each, goes at the start of the caller's message, and short tags inside the reply tell the speech model which delivery to use, which word to stress and how long to pause. The spec warns that its formats may change in incompatible ways before version 1.0.

The second change gives each of the language model's two jobs to a separate model. Think-Live decides whether the caller has finished or is only pausing, and when the should listen, wait or respond; Deepgram says it is generally available in its Voice Agent API. Think-Fast follows the business's rules, calls tools during the conversation and writes replies meant to be spoken, and is in early access on the Deepgram LLM API. In Anuff's example, a caller switches a delivery from Friday to Thursday while the agent is checking slots, and Think-Live acknowledges the change while Think-Fast updates the request. The post gives no latency or accuracy measurements for either model, and the Voice Agent Platform meant to combine Pipecat, RCS and these models has no release date yet.

Key takeaways · AI-distilled

Deepgram's Real-Time Context Standard (RCS), built with Pipecat, adds three fields to the chat completions format: how the caller sounded, what the agent should do about it, and how text-to-speech should deliver the reply.
Deepgram's CPO argues the usual speech-to-text, LLM, text-to-speech chain discards everything but the words at each handoff, so a whisper or hesitation never reaches the model and the LLM cannot tell the voice model how to speak.
Deepgram split the voice agent LLM into two models: Think-Live decides when to listen, wait, or respond (GA in the Voice Agent API), while Think-Fast follows business rules, calls tools, and writes speakable replies (early access).
Deepgram says Pipecat stays open source and model-agnostic after the acquisition, and its planned Voice Agent Platform will run Pipecat agents in production on Deepgram hosting or a customer's own cloud; timing is not yet announced.
Flux STT Multilingual now offers 10 languages with 31 more in early access, and Flux TTS is GA in nine languages including Hindi and Japanese, so where the two overlap a team can listen and reply in those languages with one provider.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Read the source deepgram.com