Hybrid (transformer–RNN) models are fast becoming a serious alternative to the transformer, but a big question remains: how do they process tokens differently & how does this impact performance?
We compared our transformer (Olmo 3) & hybrid (Olmo Hybrid) models to find out. 🧵
A transformer’s attention layers can look back at any earlier token exactly.
A hybrid model swaps most of those layers for recurrent ones that excel at sequential processing.
Do these differences give hybrid models and transformers different strengths?
To find out, we scored how well Olmo 3 & Olmo Hybrid predicted different kinds of next tokens across articles, books, papers, code, HTML, & LaTeX.
The models are matched on data, tokenizer, & training recipe, so a gap in their predictions points to architecture differences.
We found that the hybrid model advantage is highest on meaning-bearing words—the nouns, verbs, & adjectives that say what a sentence is about.
On function words like "the," "of," & "is," its advantage over transformers remains, but is more muted.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Aggregate loss hides where hybrid RNN and transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition → models differ. Filtering loss by tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → type shows transformers keep the edge on copying exactly from earlier context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, which is what long code and retrieval-style tasks lean on.
Read the full thread on X ↗
