Vibeleaderboard
← All Intel
Intel / article

RoFormer: Enhanced Transformer with Rotary Position Embedding

Source
arxiv.org
Author
Jianlin Su et al.
Date

Context

RoFormer addresses how a represents the order and spacing of text. Its Rotary Position , or RoPE, rotates the numerical representations used when compares two text positions. Each rotation depends on position, making the comparison also depend on the distance between them. This puts relative position directly into the attention calculation. The paper's derivation explains that mechanism.

It revisits a problem already present in the original Transformer, which added position information to its input representations. RoPE offers a different mathematical treatment of order; flexibility in sequence length alone does not establish reliable performance on arbitrarily long text. Original positional encoding.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
Recommended reads
Comments

Checking sign-in…

Loading comments…