OlmoEarth v1.2 swaps to RoPE to clear embedding artifacts
- Source
- x.com
- Date
Today we're releasing OlmoEarth v1.2, the latest in our family of open foundation models for Earth observation. 🌍 We've switched to rotary positional embeddings (RoPE), which reduces artifacts in the embeddings & gives a small performance boost. 🧵
OlmoEarth processes satellite images into tiles (patches), representing each as an embedding the model uses for downstream tasks—a numerical representation. Earlier versions tagged patches with a fixed position signal that surfaced as unwanted artifacts in those embeddings.
V1.2 switches to RoPE. Instead of adding a position signal to each patch, it rotates the vectors the model compares in attention by angles defined by each patch's position. Across all model sizes, we see consistent improvement on our kNN/linear-probe evals.
This update came directly from partners asking for cleaner embeddings. OlmoEarth v1.2 comes in Nano, Tiny, Small, & Base—all open source. 🤗 Models: https://t.co/9uGJLi9loK 💻 Training & fine-tuning code: https://t.co/QzncWyXuq6 📄 Tech report: https://t.co/bmeRA1zJwt
Context
Ai2's OlmoEarth v1.2 changes how a satellite-image model represents location. Earlier versions added a fixed position signal to image patches, which left unwanted patterns in the numerical representations used by downstream tasks. Version 1.2 uses rotary positional : position changes the rotation of vectors involved in comparing patches. Ai2's release thread reports reduced artifacts and illustrates the removal of banding. Following the v1.1 efficiency update, this release addresses the cleanliness of the representations themselves.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
Checking sign-in…
Loading comments…