
X-Portrait
https://github.com/bytedance/x-portrait- Category
- AI Tools
- Rank
- No. 1112Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- bytedance
- GitHub
- 541 stars
- Date
About
SIGGRAPH 2024 portrait animation model with hierarchical motion attention for expressive face reenactment from a driving video.
What it does
X-Portrait turns a source photo and motion footage into generated video. It aligns the footage to the source head pose, detects facial landmarks, isolates eye and mouth regions, then conditions a diffusion pipeline on appearance, pose, and local facial detail.
Why it's ranked here
This is compelling research code for controlled portrait generation, with unusually visible preprocessing and conditioning mechanics. The practical burden is substantial: it requires a specific Python and CUDA setup, separate pretrained weights, and careful frame matching that can affect output quality.
What's good
The pipeline automatically searches for a driving frame whose landmarks best match the source pose. It separately preserves local evidence around both eyes and the mouth. LCM LoRA compatibility can reduce inference steps, while optional results from another reenactment method can strengthen expressiveness.
Tradeoffs
Setup requires Python 3.9, CUDA 11.8, many tightly constrained dependencies, and a separately downloaded checkpoint. Face detection failures trigger fallback behavior. Output quality can depend on precisely choosing the best matching frame, so weak inputs or poor alignment may require manual iteration.
How to use it well
Use it for research experiments where you control the source portrait, driving footage, GPU environment, and sampling settings. Start with clear faces and select a close pose match before tuning frame overlap or inference steps. It does not replace a hosted API, polished editor, or complete production service.
Technical notes+
README.md specifies Python 3.9, CUDA 11.8, external checkpoints, shell-driven testing, and controls for output length, frame overlap, pose matching, and DDIM steps. requirements.txt pins a broad PyTorch, diffusion, vision, video, and face-alignment stack. core/test_xportrait.py reads images or MP4 video, detects 68-point landmarks, finds or accepts a matching pose frame, applies affine alignment, builds source, condition, and local-feature tensors, and loads checkpoints. model_lib/ControlNet/cldm/cldm.py combines temporal UNet blocks with reference attention plus pose and local-pose controls. model_lib/ControlNet/ldm/modules/attention.py implements attention-bank write and read modes, with optional xFormers memory-efficient attention.
Observed
- Primary language
- Python
- Runtime requirements
- Python 3.9 and CUDA 11.8 are required by the README.
- Installation surface
- Shell-based environment installation with dependencies declared in requirements.txt.
- Inference interface
- The README documents shell-launched generation with source image, driving video, output, frame, overlap, and sampling parameters.
- Model distribution
- Pretrained weights must be downloaded separately and placed under the checkpoint directory.
- License
- No license is stated in the supplied repository text.
Read from README.md, requirements.txt, core/test_xportrait.py, utils/utils.py, utils/__init__.py, utils/io_utils.py, utils/txt_utils.py, utils/time_utils.py, model_lib/ControlNet/ldm/util.py, model_lib/ControlNet/cldm/cldm.py, model_lib/ControlNet/cldm/model.py, model_lib/ControlNet/ldm/data/util.py, model_lib/ControlNet/ldm/modules/ema.py, model_lib/ControlNet/ldm/modules/attention.py.
What it can do
Animate portrait from driving video
Static portrait image and driving video → Animated portrait video with facial expressions and movements
Transfer facial expressions between faces
Source face video and target portrait image → Target portrait with source facial expressions applied
Generate expressive face reenactment
Reference facial motion video and target face image → Realistic animated face with transferred expressions
Extract hierarchical facial motion patterns
Video with facial movements → Structured motion data with hierarchical attention maps
Synthesize realistic portrait animations
Static portrait photo and motion reference → High-quality animated portrait video
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.