← All IntelClip / EducationHow a cascade voice assistant actually works, and why it's slow
From Full Duplex Models: Moshi and the New Voice AI Paradigm · ≈2:23
Walks the full VAD to ASR to LLM to TTS pipeline and names its two structural costs — acoustics discarded at transcription and latency accumulating across components versus a human's ~200ms.
What’s in it
- Walks the full VAD to ASR to LLM to TTS pipeline and names its two structural costs — acoustics discarded at transcription and latency accumulating across components versus a human's ~200ms.
Clip transcript
st bird cuts cuts cats. BFFs bird do do me cats. >> Okay, so what in the world is happening here? The issue is standard mode is basically an LLM wrapper. If you ask the assistant, it will tell you it was built on top of GPT 5.5, which is a pure language model that cannot process or produce audio on its own. There's an entire pipeline of additional components that gives it a voice interface. A voice activity detection or VAD model continuously analyzes the audio waveform captured by the microphone. >> Like, when did you give it to yourself? >> Well, right when you asked me if I had a name. >> Every 30 milliseconds or so, it makes a binary decision. Is the user speaking or are they silent? After long enough silence, it concludes the speaker is done talking. Then every identified speech portion is passed on to an automatic speech recognition model or ASR, which transcribes it. Next, a pure language model like GPT 5.5 reads the user input in text form and produces a text response. So, by the time the message gets to the LLM, the acoustics are lost in translation. I mean, transcription. And finally, a text-to-speech or TTS model reads the LLM response out loud. This design is representative of a large class of voice assistants known as cascades. Cascades can be quite slow since latency accumulates across components. OpenAI hasn't published latencies for GPT-5.5 in particular, but its predecessors took multiple seconds to respond with voice. For reference, a human takes a little over 200 milliseconds on average.
Comments
Sign in to comment.
Loading comments…