Vibeleaderboard
← All Intel
Intel / post

Why FP4 pretraining should use a uniform grid, not E2M1

Source
x.com
Date
Ant Ling@AntLingAGI

We recently released a paper showing that UFP4, our uniform-grid FP4 training recipe, stays closer to BF16 than strong E2M1 baselines across Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining. The key insight: FP4 training quality is not only about bit width, but also grid geometry.

Context

Ant Ling says that when a is trained in 4-bit numbers, the spacing of the values those 4 bits can represent matters alongside the bit count. FP4 training stores and multiplies numbers in 4 bits to save memory and computation, and the standard FP4 layout, E2M1, spaces its values unevenly, close together near zero and farther apart at larger sizes. The team's paper, posted June 18, 2026, argues that this uneven grid makes rounding lopsided. On average it shrinks values toward zero, an error the authors call shrinkage bias, and that error compounds from layer to layer instead of cancelling out. A Random Hadamard Transform, a step used to spread out extreme values before rounding, makes the bias worse on E2M1, which the authors suggest may explain instability in existing E2M1 training recipes.

Their recipe, UFP4, uses evenly spaced 4-bit grids (the E1M2 or INT4 formats) instead. It applies the Hadamard transform to all three matrix multiplications in a training step and uses stochastic rounding, which rounds up or down at random in proportion to distance, only for the output gradient. In long runs on a 1.5-billion-parameter dense model and 7.9-billion and 124-billion-parameter models, the authors report that UFP4 lost less quality relative to 16-bit BF16 training than strong E2M1 baselines. That matters for chip design because current FP4 hardware is built around E2M1, and the authors recommend that future accelerators support uniform 4-bit grids as first-class training formats alongside it.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
More from Ant Ling
Recommended reads
Comments

Checking sign-in…

Loading comments…