Using group theory to explore the space of positional encodings for attention
Source
blog.janestreet.com
Date
Why it matters
If you have wondered whether a better positional encoding is waiting to be found, this argues the space is nearly exhausted: the constraints leave only a few families, and almost all of them are already in use.
