Understanding Positional Embeddings in Modern LLMs

Sebastian Raschka's walkthrough of positional embeddings ends on a deliberately unexciting conclusion: RoPE, short for rotary position embedding, is still the predominant way open-weight language models keep track of word order in 2026. What has changed is how fully it is applied. More recent models rotate only part of each attention head, or skip RoPE in some layers, and one design replaces it with a learned bias. Raschka gets there by rebuilding the problem from the start.

The problem is that attention, the operation that lets each token weigh every other token, has no built-in sense of order. Raschka's example is "The red fox jumps over the gray fox." Both foxes get the same token ID and therefore the same embedding vector, and in self-attention with no positional encoding and no causal mask, swapping "red" and "gray" leaves every token's embedding and every pairwise attention score unchanged, even though a different fox now does the jumping. GPT-style models do use a causal mask, which lets each token see only itself and earlier tokens, and that gives some implicit order information. But the mask cannot tell a gap of one position from a gap of ten, which is why GPT-2 and most newer models add explicit position information.

The early answer was to add a position vector to each token's embedding before the first transformer block. GPT-2 learns a table of 1,024 such vectors during training, so it only works on inputs of 1,024 tokens or fewer, and the context length has to be fixed before training starts. Raschka calculates that in GPT-2 Medium the table makes up 0.30% of the model's weights at 1,024 positions but would exceed 22% at a context of around 100,000 tokens. The original 2017 transformer computed its position vectors from fixed sine and cosine functions instead, which can be evaluated beyond the training length, though the article notes that does not guarantee the model performs well there. Both are absolute methods: each vector describes a token's distance from the start of the sequence.

RoPE, introduced by Su et al. in the 2021 RoFormer paper, moves position inside attention and makes it relative. Each token's query and key vectors (the two projections whose dot product sets the attention score) are split into pairs of numbers, and each pair is rotated by an angle proportional to the token's position, with different pairs rotating at different speeds. Value vectors are not rotated. Because the query and the key are both rotated, their dot product depends on the difference between their angles, and so on how far apart the tokens are. In Raschka's worked example, a query at position 5 and a key at position 2 end up 60 degrees apart and one pair contributes 13 to the score. Shift both tokens three places, or put a 100-token system prompt in front of the sentence, and the contribution is still 13. Move only the key one position closer and it rises to about 19.92. Raschka adds a caveat: the score stays fixed only if the unrotated vectors stay fixed, and in a real model a new prefix can change those vectors by changing what the tokens attend to.

According to the article, RoPE has been a staple of open-weight models since the original LLaMA, and most of the newer variants trim it rather than replace it. Partial RoPE rotates only some features: MiniMax-M3 rotates 64 of the 128 dimensions in each head. Raschka puts the saving at 0.134 GB for MiniMax-M3's 1-million-token context if the precomputed sine and cosine tables are stored in BF16, while the KV cache still stores full keys and values. A paper the article cites, "Fractional Rotation, Full Potential?", found that rotating about 10% of each head's features often matched full RoPE on training loss and benchmark accuracy in 1B to 8B models, but full RoPE did better on some evaluations, and Raschka's view is that partial RoPE is hardly worth it. NoPE, meaning no explicit positional encoding, gives a layer no position signal of its own beyond the causal mask, though in a mixed model it still inherits position information from earlier layers: SmolLM3 applies RoPE in three layers and skips it in the fourth across its 36 layers, a pattern its authors said improved long-context performance. Thinky's Inkling instead adds a learned relative position bias to each attention score before softmax, based on the query token's hidden state and how far back the key is, a simpler form of which appeared in Google's T5 in 2019.

The trend claims come from Raschka's own LLM Architecture Gallery, grouped by release date. Using RoPE in every softmax attention layer is still the most common choice, but its share fell from 2025 to 2026 and again in the second half of 2026, while models that mix RoPE and NoPE layers grew. Among models that use RoPE, partial RoPE went from a minority in 2025 to a majority in both 2026 groups. Raschka cautions that the groups are small, so it is hard to say whether these trends will continue, and each gallery card counts once, so several variants of one architecture can all count. The exact shares are in charts that were not available for this summary. Raschka expects RoPE to remain predominant into 2027, possibly alongside more Inkling-like designs. The article is written as a building block for an upcoming architecture analysis, and its closing reading list points to Raschka's earlier pieces on the parts it does not cover, including a visual guide to attention variants such as grouped-query and sliding-window attention.