Intel / article

Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Source: huggingface.co
Author: Hugging Face
Date: Published Jul 23, 2026

Context

Nunchaku Lite makes compressed image-generation models usable through the ordinary Diffusers loading workflow. Its technical introduction describes replacing selected model operations with low-precision versions and downloading the required GPU kernels, avoiding a separate engine or local compilation.

The main computation uses four-bit weights and intermediate values, while a smaller, higher-precision correction preserves information that is difficult to compress. That can reduce memory use and accelerate image generation. The integration favors compatibility with standard model structure: it gives up some of the architecture-specific optimizations that make the original Nunchaku engine faster. Users gain a simpler loading path, with hardware and checkpoint compatibility still determining what will run.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.

Key quotes

“Large diffusion transformers can create stunning images (or even videos, audio snippets, and now text), but loading a modern text-to-image model in BF16 precision often requires 20-30 GB of VRAM, which puts these models out of reach of most consumer GPUs.”

“This reduces memory usage significantly, but it usually does not make inference faster, and can even add a small latency overhead.”

“This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder, and generates a 1024x1024 image in about 1.7 seconds on an RTX 5090 with a peak memory usage of about 12 GB, compared with about 24 GB for the BF16 pipeline.”

“SVDQuant handles this by moving activation outliers into the weights, representing the hardest part of each weight matrix with a small 16-bit low-rank branch, and quantizing the remaining residual to 4 bits.”

Read the source huggingface.co