Paradigma Fellows Detail FP8 and Optimizer Tricks Behind NanoGPT Records
Source
Paradigma
Author
Paradigma
Date
Key takeaways · AI-distilled
For the Modded-NanoGPT main track, Paradigma moved the MLP down-projection to FP8, but a naive version's extra passes to compute amax and quantize erased most of the gain. Their fused up-projection kernel emits the FP8 tensor and partial amax while values sit in registers.
The FP8 path uses delayed scaling: the activation scale at step t comes from the amax measured at step t-1 times a safety margin, turning quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition → into a streamingSending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.Full definition → step inside the producer kernel. The existing BF16 backward path is left unchanged.
On the Optimization Track, where only optimizer changes count, Paradigma applied Anderson/RRE-style extrapolation over recent weight vectors late in training, kept conservative: four checkpoints, run every five steps, damped, and capped at 0.1% of the parameter-vector norm.
For OpenAI's Parameter Golf (16MB checkpoint, 10-minute run), two near-free additions using 12 channels each, an attention gate before the output projection and an input-dependent Smear Gate mixing each token embedding with the prior one, earned a leaderboard spot.
The fellows say autonomy needs varied: SlowRun converged quickly and the Optimization Track came from rounds of autonomous AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → iteration, while the main track needed close human work on dataflow and kernels. Flywheel linked hypotheses to experiments and evidence.
Terms in this piece · Glossary
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Paradigma details how they set Modded-NanoGPT and Parameter Golf records: a streaming FP8 down-projection that avoids extra memory passes, a conservative late-training weight extrapolation optimizer, and near-free attention and embedding gates, all concrete techniques for speeding up LLM pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition →.
