Intel / article

From GPT-2 to gpt-oss: Analyzing the Architectural Advances

Source: magazine.sebastianraschka.com
Author: Sebastian Raschka, PhD
Date: Published Aug 9, 2025

Context

OpenAI's gpt-oss models use two different techniques to reduce resource demands. As Sebastian Raschka explains in an architecture analysis, layers contain multiple neural subnetworks but select only some for each text token. This reduces the computation needed compared with activating every expert.

Storing those subnetworks still takes memory. The separate MXFP4 technique represents their weights at lower precision, making them smaller to store. The official model card confirms that the released models use quantized expert weights. Selective activation and compact storage address different constraints, which is why parameter count alone cannot describe a model's hardware demands.

Key takeaways · AI-distilled

SwiGLU uses three weight matrices instead of two, but is smaller, not bigger, because each is roughly half-width. At 1024 embedding dim a classic feed-forward block costs about 8.4M parameters and the gated version about 3.1M.
Dropout disappeared because it solved a problem that no longer exists. It was designed for hundred-epoch training; an LLM sees each token about once, so there is little to overfit. A 2025 Pythia-1.4B study found dropout actively hurts in that regime.
RoPE (rotary position embedding) encodes order by rotating the query and key vectors by an amount that depends on each token's position, instead of adding a separate learned position vector to the embedding the way GPT-2 did.
Swish beat GELU mainly on cost, not quality. GELU needs the Gaussian error function, approximated with polynomials; Swish is just x times sigmoid(x). Reported accuracy differences between the two sit inside hyperparameter noise.
The striking thing from GPT-2 to gpt-oss is how little the base changed. Staff rotate between labs and nothing has beaten the at scale: the highest-ranked non-pure-transformer on LM Arena sits around rank 22.

Terms in this piece · Glossary

mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.

Key quotes

“The 20B model can run on a consumer GPU with up to 16 GB of RAM. The 120B model can run on a single H100 with 80 GB of RAM or newer hardware.”

“In gpt-oss, the window is just 128 tokens, which is remarkably small.”

“I am really surprised by how good their open-source models really are compared to their best product offering in terms of benchmark performance”

“By the way, I can run the gpt-oss-20b model comfortably on my Mac Mini using ollama. It uses about 13.5 Gb or memory, which is really reasonable.”

Read the source magazine.sebastianraschka.com