Clip / AI Tools

Training multimodal from step zero instead of bolting vision on

From MiniMax's Open Weight Model Strategy and Inference Stack · ≈10:45

Explains why joint pretraining avoids the collapse other labs hit and yields text tokens that attend to visual tokens — useful for RL on visual tasks like building websites.

What’s in it

Explains why joint pretraining avoids the collapse other labs hit and yields text tokens that attend to visual tokens — useful for RL on visual tasks like building websites.

Watch this moment

Clip transcript

architecture of it afterwards as well. >> Right. Definitely. So, what's different from before was that it was trained multimodal from scratch. So, from step zero, we trained not only text data, we also trained image data. And it was normal for many other labs that the model would collapse after training a little bit, and we managed to solve that problem. And what we found was actually that with this kind of training from scratch, if you look at the attention map, it actually the visual of the the text tokens would attend to the visual tokens. So, that they are naturally combined together, they naturally understand each other. So, for example, we are developing, for example, websites, right? It is better if we train with both modalities. Also, like, for example, it can look at the website, it can understand how it looks, and then better optimize for it. Like for example, during reinforcement learning. Um so yeah, I think that is pretty cool.