← All IntelClip / EducationAutoregressive versus diffusion for music and audio generation
From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈1:17:29
A practitioner's read on a real architecture choice: both approaches ship, but next-token prediction is awkward for abstract continuous modalities like music.
What’s in it
- A practitioner's read on a real architecture choice: both approaches ship, but next-token prediction is awkward for abstract continuous modalities like music.
Clip transcript
on each other? Uh when you might want to like generate everything at the same time. Y- Y- You can do both. There's music models that are auto regressive, there's music models that are diffusers. Uh like it depends on how you how you train it. Um Like I think some of the some of the Google models are like like transformer based. Some of uh open source models are like diffuser based. Like both kind of work kind of work very well. It's just that this it's a little bit As I said, like it's a little bit more difficult to uh to to put into perspective this the concept of like all music. If you tokenize and predict the next token, it's kind of hard cuz it's very abstract. Uh so usually diffusers work a bit better in this like image modalities for generation or like music or like even audio for some models. Like they have some kind of diffuser diffuser um diffuser part of the process. Um both can work. Diffusers are generally a bit easier to get it working. I hope this answered your question.
Comments
Checking sign-in…
Loading comments…