Intel / article

Now in Nature: Retrofitting language models to operate over bytes

Source: allenai.org
Date: Published Oct 7, 2026

Context

Most language models split text into subwords, chunks from a fixed vocabulary. Byte-level models read the bytes computers use to store characters, but matching subword models has historically required training from scratch. Ai2 says in its October 7, 2026 post that its Bolmo work, now published in Nature, uses a process it calls byteifying: converting a capable subword model with a relatively short additional training run.

The new result is that this extends beyond Ai2's own Olmo models. Applied to Qwen 3 8B and Llama 3 8B, it produced Bwen 8B and Blama 8B, which Ai2 says come close to the models they were derived from. Bwen 8B outperforms Bolmo 7B on Ai2's aggregate suite. Ai2 also released stage 1 checkpoints, which keep the original model frozen while the new byte-level components train.

These are Ai2's own comparisons, and its hopes for images and audio are possibilities, not results.

Key takeaways · AI-distilled

Ai2's "byteifying" converts an existing subword model into a byte-level one with a relatively short extra training run, avoiding the from-scratch training that Ai2 says byte-level models historically needed to match subword performance.
Applying the recipe to Qwen 3 8B and Llama 3 8B produced Bwen 8B and Blama 8B. Ai2 reports both come close to their source models, and Bwen 8B outperforms Bolmo 7B on Ai2's aggregate evaluation suite.
Ai2 argues byte-level input helps with spelling, rare words, unusual strings, whitespace and varied writing systems, while follow-up research is probing the tradeoff between hierarchical byte architectures' efficiency and fine-grained character understanding.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source allenai.org