RoFormer: Enhanced Transformer with Rotary Position Embedding
Source
Jianlin Su et al.
Author
Jianlin Su et al.
Published
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Rotary embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → are why modern models handle long contexts at all, and why context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →-extension tricks work the way they do: position is applied as a rotation, so relative distance falls out of the dot product and the scheme can be stretched after training.
