Self-Flow trains image, video and audio generation without external encoders
- Source
- x.com
- Date
We present a research preview of Self-Flow: a scalable approach for training multi-modal generative models. Multi-modal generation requires end-to-end learning across modalities: image, video, audio, text - without being limited by external models for representation learning. Self-Flow addresses this with self-supervised flow matching that scales efficiently across modalities. Results: • Up to 2.8x faster convergence across modalities. • Improved temporal consistency in video • Sharper text rendering and typography This is foundational research for our path towards multimodal visual intelligence.
Self-Flow improves temporal consistency in video generation. 4B parameter multi-modal model trained on 6M videos.
Joint video-audio generation from a single model (sound on) 4B parameter multi-modal model trained on 2M audio-video pairs.
Context
Self-Flow trains a media-generating model to learn useful internal patterns without borrowing them from a separately pretrained feature teacher. In Black Forest Labs' method, the learner receives data with uneven amounts of added noise. A slowly updated copy of the model sees a cleaner version, and the learner tries to reproduce that copy's internal representation while also learning to generate the data. The paper explains this paired training task in its method section.
The researchers report that Self-Flow converges up to about 2.8 times faster than REPA, an approach that relies on an external teacher. That result concerns training progress, rather than the speed of generating an image for a user. It supports a way to improve training without a separate feature teacher; the experiments still use autoencoders to compress the input data. The paper's experiments and implementation details describe these distinctions.
Checking sign-in…
Loading comments…