LatentSync
https://github.com/bytedance/latentsync- Category
- AI Tools
- Rank
- No. 1031Tools index
Previous survey · No. 1012 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- bytedance
- GitHub
- 6.1k stars
- Date
About
ByteDance's diffusion-based lip-sync model — tame Stable Diffusion to synchronize mouth movements with arbitrary audio.
What it does
LatentSync takes an existing video and separate audio track, then rebuilds facial motion so speech and mouth movement align. Whisper-derived audio features guide a video U-Net through cross-attention, while reference frames and masked frames preserve the surrounding visual context.
Why it's ranked here
This is a serious research implementation, not merely a demo wrapper. It includes checkpoints, inference, data preparation, training, and evaluation. The main reservation is operational cost: even inference needs substantial GPU memory, while full training configurations demand much more.
What's good
The repository covers the complete workflow from raw-video filtering through model training and output evaluation. Users get both a browser interface and command-line inference. Adjustable sampling steps trade speed for visual quality, while guidance strength trades synchronization accuracy against possible distortion or jitter.
Tradeoffs
The lighter checkpoint requires at least 8 GB of VRAM, while the higher-resolution checkpoint requires 18 GB. Training ranges from 20 GB for the efficient second stage to 55 GB for high-resolution second-stage training. Evaluation data must pass through the project’s specialized preprocessing, limiting direct comparisons with untouched datasets.
How to use it well
It best suits researchers and technical video teams that already have source footage, replacement audio, and CUDA hardware. Start with the browser interface for visual checks, then use scripted inference for repeatable batches. Treat guidance and sampling settings as quality controls. It does not create the source performance or replacement audio.
Technical notes+
scripts/inference.py loads a DDIM scheduler, Stable Diffusion VAE, Whisper audio encoder, 3D conditional U-Net, and optional DeepCache helper before invoking the lip-sync pipeline on CUDA. gradio_app.py wraps the 512-resolution configuration with video, audio, guidance, step, and seed controls. predict.py provides a Cog prediction interface and downloads packaged weights when absent. scripts/train_unet.py implements distributed mixed-precision training with SyncNet, LPIPS, and TREPA supervision. preprocess/sync_av.py filters and offsets clips through multi-GPU SyncNet processing. eval/fvd.py and eval/eval_fvd.py calculate feature-distribution distance, while eval/eval_sync_conf.py reports synchronization confidence and audiovisual offset. requirements.txt pins the Python and CUDA-oriented dependency stack.
Observed
- License notices
- Supplied Python source files carry Apache License 2.0 headers.
- Primary language
- Python
- Installation surface
- A shell setup command installs pinned requirements and downloads model checkpoints; manual Hugging Face checkpoint download is also documented.
- Interfaces
- Gradio browser application, shell-driven command-line inference, and Cog prediction interface.
- Hardware support
- CUDA GPU execution is required by the inference pipeline; documented minimum VRAM is 8 GB or 18 GB depending on checkpoint.
- Project scope
- Repository text includes inference, data processing, U-Net training, SyncNet training, and evaluation workflows.
Read from README.md, requirements.txt, docs/syncnet_arch.md, docs/changelog_v1.5.md, docs/changelog_v1.6.md, predict.py, gradio_app.py, eval/fvd.py, eval/eval_fvd.py, eval/hyper_iqa.py, tools/occupy_gpu.py, scripts/inference.py, preprocess/sync_av.py, scripts/train_unet.py, eval/eval_sync_conf.py.
What it can do
Synchronize lip movements with audio
Video file and audio file → Video with lip-synced mouth movements
Generate realistic mouth movements from speech
Audio speech file → Facial animation data with synchronized lip movements
Process arbitrary audio for lip synchronization
Any audio file containing speech → Mouth movement parameters matching the audio
Apply diffusion-based lip sync to existing videos
Video with faces and replacement audio track → Modified video with new lip-synced dialogue
Create lip-synced animations using Stable Diffusion
Audio file and face image or video → Diffusion-generated video with synchronized mouth movements
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.