Caramel: a speech-to-speech conversation model trained only on audio

caramel-1 is a small, open speech-to-speech conversation model that the developer naklecha trained and released with its code, training recipe and weights. It learned only from recordings of people talking: no transcripts or other text were used, so the model "has never seen a single word of text," in the words of the write-up. The author says it "sounds incredibly human" but that "most of what caramel-1 says doesn't make sense yet." That split is the useful way to read the project. On the author's own measurements it comes close to real people on several timing measures, while most of what it says still does not make sense.

The model is full duplex, meaning it listens and speaks at the same time instead of waiting for the other person to finish. It has 413 million parameters and two input streams, its own voice and the other speaker's. A neural audio codec first turns each speaker's audio into tokens, 12.5 frames a second with 8 tokens per frame. Every 80 milliseconds a larger transformer reads everything both people have said so far, and a smaller one writes the tokens for the next frame. There is no voice activity detector, the component many voice systems use to notice when someone is speaking, and no turn-taking rule. The model predicts when to talk the same way it predicts the sound. Because every training conversation was used once from each speaker's side, the same weights can play either person, which is how two copies of caramel-1 hold a conversation with each other in the published samples.

The project was inspired by Dessa, a speech-to-speech model from a small research lab at Block that was also trained on human speech rather than text. naklecha measured 13 statistics on Dessa's example conversations, such as how often people overlap, how long the gaps between turns are and how many backchannels (the "mm-hm"s listeners make) occur per minute, and used how close a source came to those numbers to decide which public podcasts and Creative Commons YouTube channels to download. The training set came to 5,743 hours of two-person conversation and 5,812 hours of one-person speech such as audiobooks. Training took about 2 hours 15 minutes, roughly 4 GPU-hours, and the author reports that more hours and more passes over the data lowered the model's prediction error on unseen conversations and made its words clearer to a speech recogniser.

On 100 conversations it never trained on, the author's table puts caramel-1 close to real people on some measures: both speakers talking at once 8.8% of the time against 9.0%, and 191 words per minute against 190. Others are further off. It changes turns 8.3 times a minute against 10.2, makes 2.5 backchannels a minute against 3.5, and its median gap between turns is minus 56 milliseconds, meaning the next speaker usually starts before the last one finishes, against 105 milliseconds for real people. An automatic naturalness scorer called UTMOS rated both at about 2.5 on its 1 to 5 scale. The same table also shows a classifier trained to spot generated audio, which, in the author's words, "can still easily tell the model apart from real people." The author also reports speed: on a GPU the model makes 1 second of audio in about 0.09 seconds, and on an M3 Pro MacBook Pro two copies together make 1 second of conversation in about 0.25 seconds. These are all the author's own measurements, none of them test whether the words make sense, and the write-up presents caramel-1 as "a small experiment I ran in a few hours" to demonstrate Cloudy, the research platform the author founded and used to build it.