• SwiGLU uses three weight matrices instead of two, but is smaller, not bigger, because each is roughly half-width. At 1024 embedding dim a classic feed-forward block costs about 8.4M parameters and the gated version about 3.1M.
• Dropout disappeared because it solved a problem that no longer exists. It was designed for hundred-epoch training; an LLM sees each token about once, so there is little to overfit. A 2025 Pythia-1.4B study found dropout actively hurts in that regime.
• RoPE (rotary position embedding) encodes order by rotating the query and key vectors by an amount that depends on each token's position, instead of adding a separate learned position vector to the embedding the way GPT-2 did.
• Swish beat GELU mainly on cost, not quality. GELU needs the Gaussian error function, approximated with polynomials; Swish is just x times sigmoid(x). Reported accuracy differences between the two sit inside hyperparameter noise.
• The striking thing from GPT-2 to gpt-oss is how little the base changed. Staff rotate between labs and nothing has beaten the transformer at scale: the highest-ranked non-pure-transformer on LM Arena sits around rank 22.

• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
• quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
• grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
• transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.

If you're evaluating or building on OpenAI's gpt-oss models, this walks through their exact architectural choices (MoE, sliding-window attention, MXFP4 quantization) with clear comparisons to GPT-2 and Qwen3, giving you a grounded reference for how these open-weight models actually work.

• articleA Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026
• articleBeyond Standard LLMs
• articleUnderstanding the 4 Main Approaches to LLM Evaluation (From Scratch)
• articleUnderstanding and Implementing Qwen3 From Scratch

• articleThe Big LLM Architecture ComparisonSebastian Raschka, PhD
• articleA Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026Sebastian Raschka, PhD
• videoGPT-5.6Bijan Bowen

Checking sign-in…

Loading comments…