Hybrid vs transformer: per-token loss shows where each wins

Ai2 compared its transformer Olmo 3 with Olmo Hybrid one predicted token at a time, and found the hybrid's advantage concentrated on words that carry meaning and nearly absent when text is copied from earlier in the document. The two models were built to be closely matched in data, tokenizer and training recipe, so differences mostly reflect architecture. A transformer's attention layers let every position look back at any earlier token directly. The hybrid keeps a few attention layers and replaces the rest with recurrent layers, which read left to right and carry a fixed-size running memory. For each token, the researchers recorded how much probability each model gave the actual next token and compared the two models' losses. Ai2's write-up.

The hybrid had lower loss on most kinds of token, with a gap of about 0.04 on content words such as nouns, verbs and adjectives and about 0.02 on function words such as "the" and "of." It also did better on probes that require remembering which person a pronoun refers to. Its lead shrank on repeated passages, approaching zero as the repeated run grew longer, and it was smaller on closing brackets than opening ones, while the transformer did better on bracket-matching probes. The authors suggest recurrent layers help with predictions that depend on the document's overall meaning so far, while attention handles exact copying and bracket matching. They present this as a suggestion, not a demonstrated mechanism. Technical report.

The practical point is about measurement. A single average loss would rank the hybrid ahead and hide where it gains and where it does not. In a proof of concept with three 1-billion-parameter models, a pure recurrent model with no attention fell behind both others on repeated tokens, since it has nothing to copy from. Ai2 argues that losses filtered by token type can reveal such architecture differences early in training. The findings cover these specific Olmo models. Only the opening post of Ai2's thread was collected, and its attached chart was not inspected, so the figures here come from Ai2's blog post and report abstract. Filtered evaluations.