PowLU tames SwiGLU's quadratic tail to stabilize FP8 pretraining
- Source
- x.com
- Date
SwiGLU is everywhere in modern LLMs — but for large inputs it behaves like x². That quadratic blow-up inflates activations, amplifies outliers, and makes deep network or low-precision (FP8/FP4) training prone to loss spikes. We propose PowLU, a drop-in activation built for stable large-scale pre-training. 🧵

Context
An activation function is the small formula inside each layer of a that bends its numbers so the network can learn more than straight lines. Ant Ling argues that SwiGLU, an activation used throughout modern language models, grows roughly with the square of its input once that input gets large. Squared growth turns a few big values into very big ones, and those outliers are what make deep networks and low-precision training (FP8 or FP4, number formats that store fewer digits to save memory and compute) prone to loss spikes, sudden jumps in training error that can derail a run. Its proposed replacement, PowLU, keeps SwiGLU's shape for inputs at or below zero and swaps the positive side for a rational power function, which the Ling Team's paper says brings large-input growth down to roughly linear.
The paper's evidence is that the gentler curve costs little and spikes less. In scaling-law runs on models from 26 million to 368 million active parameters, the authors report that the SwiGLU and PowLU loss curves almost overlap, and on Ling models with 7.9 billion and 124 billion total parameters they call PowLU competitive with SwiGLU and a clipped SwiGLU variant, with SwiGLU still ahead on some benchmarks. In their FP8 test, the clipped variant hit a loss spike around step 77,000 while PowLU stayed near a loss of 1.32. That comparison is narrower than it sounds: the alternative activations were swapped in partway through an existing run rather than trained from scratch, so it shows PowLU riding through the stretch where the clipped variant spiked, not a full FP8 training run built on PowLU.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
postAnt Ling releases six Ling-3.0 base checkpoints for continued pretrainingAnt Ling
articlePretraining and adapting a language model on a dependency-free stack: GPT-2 124M from random weights, reproduced against llm.c, and a clinical adapter for Qwen3-0.6BThang Tran (CloudKites AI Lab, New South Wales, Australia), Lan Dang (Monash Business School, Monash University, Victoria, Australia)
postFused MoE Pallas kernel halves Ling-2.6-1T prefill latency on TPU v7xAnt Ling
Checking sign-in…
Loading comments…



