Intel / article

Pretraining and adapting a language model on a dependency-free stack: GPT-2 124M from random weights, reproduced against llm.c, and a clinical adapter for Qwen3-0.6B

Source: arxiv.org
Author: Thang Tran (CloudKites AI Lab, New South Wales, Australia), Lan Dang (Monash Business School, Monash University, Victoria, Australia)
Date: Published Sep 25, 2026

Context

Numbat is a machine-learning stack written in Zig whose authors report reproducing a complete small language-model training run without third-party runtime dependencies. They trained a 124.4-million-parameter GPT-2 from random starting weights and compared it with a reference implementation.

The resulting held-out prediction loss was close to the published reference result. That supports the feasibility of a separate training implementation, while the authors caution that independently written versions do not fail independently, so agreement cannot rule out mistakes they share. The model weights, a configuration and an curve are released, but the numbat framework source is not, limiting inspection of the implementation itself.

Key takeaways · AI-distilled

The stack, numbat, is written in Zig with no third-party runtime dependencies. It pretrained a 124.4M-parameter GPT-2 from random weights on 9.91B of web text, with a reference implementation run on identical hardware.
Results track published figures: held-out cross-entropy of 3.2588 against a published 3.29, and HellaSwag of 0.3053 against 0.299. Across 8 paired evaluations it came in below a same-machine reference every time, by 0.0608 on average.
To separate method from framework, the authors reran the reference on different hardware; it moved by only 0.0035, which bounds how much of any gap comes from hardware.
In one measured session numbat reached 43,374 tokens/s against PyTorch's 41,202, scaling 2.769x over three cards. Weights are released under Apache-2.0, but the framework source is not.
A clinical question-answering adapter for Qwen3-0.6B ended at 2.1899 held-out loss against 2.1941. The authors state neither model is a medical device or validated for clinical use.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source arxiv.org