Paradigma Details FP8 and Optimizer Tricks Behind NanoGPT Records

Eight-bit arithmetic only speeds up model training if preparing the numbers does not cost more memory traffic than the arithmetic saves. That is the lesson of the most detailed section of a retrospective by Paradigma's Via Panisperna fellows, who say their work set records on several public training challenges, including the Modded-NanoGPT Main Track. That track asks entrants to train a GPT-2-scale model to a fixed validation loss in the shortest wall-clock time. The post explains methods but does not state the times or margins of its records.

The fellows moved the MLP down-projection, one of the most expensive matrix multiplications in each transformer block, to FP8, an eight-bit floating-point format whose matrix multiplications are theoretically faster than those in the 16-bit BF16 format. Converting values to FP8 requires a scale set by the largest absolute value in the tensor. Done naively, that means writing out the whole intermediate tensor, scanning it once to find that maximum and reading it again to convert it, and the fellows say those extra passes erase most of the theoretical speedup. Their version does the work while the preceding computation still holds the values in fast registers: one fused kernel writes the BF16 copy needed for the backward pass, emits the FP8 copy and keeps small running maxima. To avoid waiting on the current step's maximum, it uses the previous step's value multiplied by a safety margin. The large tensor is never reread, which made a forward-only FP8 down-projection profitable end to end, while the backward pass stayed unchanged.

The fellows also report on working with coding agents across these challenges. They began each one with broad exploration by several agents in parallel, tracked hypotheses, experiments and results in Paradigma's Flywheel research graph, and stepped in more as promising directions emerged. The amount of human work differed sharply. The Optimization Track record, which extrapolates the model's weights from the last four checkpoints and caps each jump at 0.1% of the weights' overall size, came from several rounds of autonomous iteration. The Main Track FP8 work needed close human involvement down to kernels and dataflow. Their conclusion is that different problems need different levels of agent autonomy, and that knowing when and at what level to intervene is a skill in its own right.