If you have wondered whether a better positional encoding is waiting to be found, this argues the space is nearly exhausted: the constraints leave only a few families, and almost all of them are already in use.

• articleRoFormer: Enhanced Transformer with Rotary Position EmbeddingJianlin Su et al.
• clipEncoder and decoder are redundant; later models pick oneJulia Turc
• articleAttention Is All You NeedAshish Vaswani et al.

Checking sign-in…

Loading comments…