← All IntelClip / EducationTool calling in a real-time voice model: two models, not one
From Full Duplex Models: Moshi and the New Voice AI Paradigm · ≈17:17
Describes the architecture both OpenAI and Thinking Machines confirmed — the speech-to-speech model delegates to an asynchronous background LLM so multi-second tool calls never stall the conversation.
What’s in it
- Describes the architecture both OpenAI and Thinking Machines confirmed — the speech-to-speech model delegates to an asynchronous background LLM so multi-second tool calls never stall the conversation.
Clip transcript
Weekly, but it looks like you haven't worked there in many years. >> What Samantha just did here is tool calling. She had to call the Gmail API to access all those emails. Both OpenAI and Thinking Machines have confirmed that they can already handle this. The core speech-to-speech model delegates to an asynchronous background model, presumably a powerful LLM, while keeping the conversation going at the same time. >> So, agent is going to do a lot of processing in the background. It's going to do tool calling. It's going to reason. It's going to do web searches and so on. But you don't want to just wait while all of this is done, right? It's going to take seconds to dozens of seconds. What you want is to keep a natural flow of conversation that is kind of asynchronous with the computations that happens in the background. So, that's something we we introduced with Moshi RAGs and this also is used by the interaction model from Thinking Machines. So, instead of having one model, you have two models.
Comments
Sign in to comment.
Loading comments…