← All IntelClip / EducationOne architecture, three behaviors, and a rounding-error training budget
From Full Duplex Models: Moshi and the New Voice AI Paradigm · ≈14:38
Shifting which stream is delayed turns the same speech-to-speech model into ASR or TTS, and the training cost plus MacBook-quantized inference shows full duplex is not gated on frontier-scale compute.
What’s in it
- Shifting which stream is delayed turns the same speech-to-speech model into ASR or TTS, and the training cost plus MacBook-quantized inference shows full duplex is not gated on frontier-scale compute.
Clip transcript
fine-tuned with the rest of the weights on speech data. Now, at its core, Moshi is a speech-to-speech architecture. But, the inner monologue makes it more versatile. If we delay the text stream by a couple of seconds, the model learns to transcribe the audio it just heard, making it a speech-to-text model. The other way around, if we delay the audio stream, we can train a text-to-speech model. Of course, at inference time, the TTS model will read user text rather than its own inner monologue. So, with a single model architecture, we get three different behaviors. Moshi really paved the new way for voice models. It was trained on about 100 H100 GPUs for about a month, which might sound like a lot, but for Frontier Labs, that's just a rounding error. Yet, Moshi responds faster than the average human, and if you quantize it, it even runs on your MacBook. So, if full duplex models
Comments
Sign in to comment.
Loading comments…