SwiGLU is everywhere in modern LLMs — but for large inputs it behaves like x². That quadratic blow-up inflates activations, amplifies outliers, and makes deep network or low-precision (FP8/FP4) training prone to loss spikes.
We propose PowLU, a drop-in activation built for stable large-scale pre-training. 🧵
The idea: replace the quadratic tail with a rational power function.
For x>0, PowLU = x · x^(m/(√x + 1)) · σ(x). As x grows, the exponent → 1, so growth smoothly decays from quadratic toward linear — reining in outliers while keeping nonlinearity. For x≤0 it's identical to SwiGLU.
One hyperparameter (m). We use m=3.
Does the smoother tail cost you anything? Almost nothing.
Scaling-law fits for SwiGLU vs PowLU nearly overlap (exponents -0.0659 vs -0.0661) from 26M to 368M activated params.
Across 17 benchmarks at 7.9B (600B tokens) and 124B (800B tokens), PowLU stays competitive — sometimes ahead.
Where it really shows: FP8 training stability.
SwiGLU and SwiGLU-Clip both hit loss spikes around step ~77k. PowLU holds a smooth trajectory near 1.32 throughout, with visibly tighter activation & gradient ranges and far fewer outlier channels.
Stability without giving up expressiveness.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
If FP8 pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → runs spike, the activation function may be the cause. PowLU keeps SwiGLU's scaling behavior while bounding tail growth, and held a smooth loss curve where SwiGLU and SwiGLU-Clip diverged.
Read the full thread on X ↗
