tufazip: Learning to Compress a Billion Bytes of Wikipedia
Source
tufalabs.ai
Date
Why it matters
Shows a single transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition →, trained online and seeded from a 4.33 MB offline prior, can approach hand-mixed compressors on 1 GB of Wikipedia. Ablations on replay, vocabulary size and model size give practitioners concrete tradeoffs for compression-as-evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →.
Key takeaways · AI-distilled
tufazip compressed the billion-byte enwik9 Wikipedia file to 105.39 MB including its decoder and offline weights, taking #2 on Matt Mahoney's Large Text Compression benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → at submission (Sept. 7, 2026) and the best entry using a neural network as its only predictor.
The encoder and decoder start from the same weights and make the same deterministic training update after every tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →, so the decoder rebuilds the model's online learning as it decodes and the weights learned online never have to be stored in the archive.
Replay, extra training on already-decoded text, helped consistently but costs compute: widening the window from 15M to 35M tokens cut the enwik9 payload from 107.22 to 105.92 MB (1.21%) for about 40% more estimated model FLOPs.
A 4K word vocabulary beat 8K and 16K, and dedicated XML/wiki tokens made compression worse despite the heavy markup. More parameters did not reliably help: on enwik9 a 267M model produced slightly larger payloads than a 188M one.
An offline prior must save more payload than its own size. Two shared blocks stored once (4.33 MB) and copied into 12 layers netted 0.59 MB over pure online learning, while a six-layer prior predicted better but its 10.9 MB decoder-and-weights bundle made the total larger.
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.