Vibeleaderboard
← All Intel
Intel / post

PowLU tames SwiGLU's quadratic tail to stabilize FP8 pretraining

Source
x.com
Date
Ant Ling@AntLingAGI

SwiGLU is everywhere in modern LLMs — but for large inputs it behaves like x². That quadratic blow-up inflates activations, amplifies outliers, and makes deep network or low-precision (FP8/FP4) training prone to loss spikes. We propose PowLU, a drop-in activation built for stable large-scale pre-training. 🧵

Context

An activation function is the small formula inside each layer of a that bends its numbers so the network can learn more than straight lines. Ant Ling argues that SwiGLU, an activation used throughout modern language models, grows roughly with the square of its input once that input gets large. Squared growth turns a few big values into very big ones, and those outliers are what make deep networks and low-precision training (FP8 or FP4, number formats that store fewer digits to save memory and compute) prone to loss spikes, sudden jumps in training error that can derail a run. Its proposed replacement, PowLU, keeps SwiGLU's shape for inputs at or below zero and swaps the positive side for a rational power function, which the Ling Team's paper says brings large-input growth down to roughly linear.

The paper's evidence is that the gentler curve costs little and spikes less. In scaling-law runs on models from 26 million to 368 million active parameters, the authors report that the SwiGLU and PowLU loss curves almost overlap, and on Ling models with 7.9 billion and 124 billion total parameters they call PowLU competitive with SwiGLU and a clipped SwiGLU variant, with SwiGLU still ahead on some benchmarks. In their FP8 test, the clipped variant hit a loss spike around step 77,000 while PowLU stayed near a loss of 1.32. That comparison is narrower than it sounds: the alternative activations were swapped in partway through an existing run rather than trained from scratch, so it shows PowLU riding through the stretch where the clipped variant spiked, not a full FP8 training run built on PowLU.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
More from Ant Ling
Recommended reads
Comments

Checking sign-in…

Loading comments…