Why parameter count alone can't tell you if a model is well-scaled

Jie Tang uses the GLM-5.3 release from Z.ai to argue that parameter count is the wrong headline number for a new model. In his essay on scaling laws, he says GLM-5.3 has the same base model, architecture, and total and activated parameter counts as GLM-5.2. What changed, by his account, was one month of post-training: scaling long-horizon environments and reinforcement learning on top of the existing model. He calls the release a controlled experiment and says the gains are not marginal.

His case starts with how the field's rule of thumb has moved. Tang recounts that Kaplan et al. (2020) fit a scaling exponent that told labs to grow parameters faster than training data, roughly 2.7 to 1, and that GPT-3, Gopher and MT-NLG followed it. Hoffmann et al. (2022), testing about four hundred models, put the compute-optimal balance closer to 20 training tokens per parameter, with model size and data growing at the same rate. Tokens per parameter is how much training text a model sees relative to its size. Once a model is called billions of times a day, Tang argues, inference cost moves the answer again, toward smaller models trained far longer. He cites Llama-2-7B at roughly 290 tokens per parameter and Gemma-2-9B at roughly 889.

The step that matters for GLM concerns mixture-of-experts (MoE) models, which have a large total parameter count but activate only part of it at a time. Tang says total parameters roughly govern how much a model can hold, such as facts and long-tail knowledge, while activated parameters and effective depth roughly govern how many steps of a causal chain it can carry before it comes apart. He cites Roberts et al. (2025) finding that the best tokens-per-parameter ratio depends on the task, with memorization favoring more parameters and reasoning favoring more data, and unnamed follow-up MoE work reporting that at a fixed ratio, raising total parameters degrades reasoning while activating more experts helps. His target task is finding software vulnerabilities, which he describes as carrying a twenty-step chain of inference to the end rather than recalling more known CVEs.

Tang's conclusion is that total parameters matter up to a threshold, enough "to hold the world," after which more capability comes from scaling elsewhere: effective depth per forward pass and, above all, post-training. The release supports less than that. If his description of GLM-5.3 is accurate, it shows that post-training alone can produce substantial gains at a fixed size. It does not show where the threshold lies, and the post gives no figures for the gains. Z.ai's GLM-5.3 model card, covered separately, is where the same-base comparison with GLM-5.2 is put in numbers. Tang himself says base model size, pretraining data and compute per forward pass are all still on the table, and that Z.ai will come back to each.