We recently released a paper showing that UFP4, our uniform-grid FP4 training recipe, stays closer to BF16 than strong E2M1 baselines across Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining.
The key insight: FP4 training quality is not only about bit width, but also grid geometry.
The problem we investigate is Shrinkage Bias in E2M1-based FP4 training.
Because E2M1 uses a non-uniform 4-bit grid, its asymmetric RTNE rounding bins can systematically shrink magnitudes toward zero instead of behaving like zero-mean quantization noise.
This bias matters during pretraining because small negative rounding errors can accumulate across GEMMs and layers.
RHT helps reduce outliers, but with E2M1 it can also move tensors into a local-resolution-limited regime where this shrinkage effect becomes more visible.
UFP4 addresses this by using a uniform E1M2/INT4-style grid.
With the grid-level bias removed, RHT can be applied more broadly across training GEMMs, improving quantization quality while keeping the recipe practical for FP4 pretraining.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
For FP4 training, grid geometry matters as much as bit width: E2M1 rounding systematically shrinks magnitudes across GEMMs, while a uniform grid keeps loss closer to BF16 from 1.5B dense up to 124B mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → runs.
Read the full thread on X ↗
