Vibeleaderboard
← All Intel
Intel / article

Dust: Pretraining Transformers Without Backpropagation

Source
qlabs.sh
Author
E-Reverance
Date

Context

A write-up on qlabs.sh describes Dust, a method for training a small without backpropagation, the standard approach that passes error signals backward to compute each weight's update. Dust instead adds random noise to each layer's output at every , rewards the noise by how much it changed the loss (prediction error), and averages the reward-weighted noise into an update estimate. Each token acts as a separate trial, so one forward pass evaluates many perturbations, a 'virtual population' in the authors' words.

Tests used models of 2 to 243 million parameters trained on FineWeb, a web-text dataset, for up to 20 million tokens. At 100,000 and 1 million tokens Dust's final loss ends below backpropagation's (lower is better); at 10 million its fitted limit lands just above. At 20 million an extrapolated fit puts Dust below again, but the authors call it loosely constrained, not a measured limit. They say Dust is not yet compute-efficient enough to replace backpropagation.

Terms in this piece · Glossary
  • transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Recommended reads
Comments

Checking sign-in…

Loading comments…