Vibeleaderboard
Index / tool
Visit github.com
Category
AI Tools
Rank
No. 1112Tools index
Pricing
Open Source
Type
TOOL
Builder
bytedance
GitHub
541 stars
Date

About

SIGGRAPH 2024 portrait animation model with hierarchical motion attention for expressive face reenactment from a driving video.

What it does

X-Portrait turns a source photo and motion footage into generated video. It aligns the footage to the source head pose, detects facial landmarks, isolates eye and mouth regions, then conditions a diffusion pipeline on appearance, pose, and local facial detail.

Why it's ranked here

This is compelling research code for controlled portrait generation, with unusually visible preprocessing and conditioning mechanics. The practical burden is substantial: it requires a specific Python and CUDA setup, separate pretrained weights, and careful frame matching that can affect output quality.

What's good

The pipeline automatically searches for a driving frame whose landmarks best match the source pose. It separately preserves local evidence around both eyes and the mouth. LCM LoRA compatibility can reduce inference steps, while optional results from another reenactment method can strengthen expressiveness.

Tradeoffs

Setup requires Python 3.9, CUDA 11.8, many tightly constrained dependencies, and a separately downloaded checkpoint. Face detection failures trigger fallback behavior. Output quality can depend on precisely choosing the best matching frame, so weak inputs or poor alignment may require manual iteration.

How to use it well

Use it for research experiments where you control the source portrait, driving footage, GPU environment, and sampling settings. Start with clear faces and select a close pose match before tuning frame overlap or inference steps. It does not replace a hosted API, polished editor, or complete production service.

Technical notes+

README.md specifies Python 3.9, CUDA 11.8, external checkpoints, shell-driven testing, and controls for output length, frame overlap, pose matching, and DDIM steps. requirements.txt pins a broad PyTorch, diffusion, vision, video, and face-alignment stack. core/test_xportrait.py reads images or MP4 video, detects 68-point landmarks, finds or accepts a matching pose frame, applies affine alignment, builds source, condition, and local-feature tensors, and loads checkpoints. model_lib/ControlNet/cldm/cldm.py combines temporal UNet blocks with reference attention plus pose and local-pose controls. model_lib/ControlNet/ldm/modules/attention.py implements attention-bank write and read modes, with optional xFormers memory-efficient attention.

Observed

Primary language
Python
Runtime requirements
Python 3.9 and CUDA 11.8 are required by the README.
Installation surface
Shell-based environment installation with dependencies declared in requirements.txt.
Inference interface
The README documents shell-launched generation with source image, driving video, output, frame, overlap, and sampling parameters.
Model distribution
Pretrained weights must be downloaded separately and placed under the checkpoint directory.
License
No license is stated in the supplied repository text.

Read from README.md, requirements.txt, core/test_xportrait.py, utils/utils.py, utils/__init__.py, utils/io_utils.py, utils/txt_utils.py, utils/time_utils.py, model_lib/ControlNet/ldm/util.py, model_lib/ControlNet/cldm/cldm.py, model_lib/ControlNet/cldm/model.py, model_lib/ControlNet/ldm/data/util.py, model_lib/ControlNet/ldm/modules/ema.py, model_lib/ControlNet/ldm/modules/attention.py.

What it can do

  • Animate portrait from driving video

    Static portrait image and driving videoAnimated portrait video with facial expressions and movements

  • Transfer facial expressions between faces

    Source face video and target portrait imageTarget portrait with source facial expressions applied

  • Generate expressive face reenactment

    Reference facial motion video and target face imageRealistic animated face with transferred expressions

  • Extract hierarchical facial motion patterns

    Video with facial movementsStructured motion data with hierarchical attention maps

  • Synthesize realistic portrait animations

    Static portrait photo and motion referenceHigh-quality animated portrait video

Tags

portraitanimationsiggraphdiffusionbytedance

Tech Stack

Python

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.