RTEB uses held-out datasets to test embeddings that overfit MTEB
- Source
- x.com
- Date
To evaluate embeddings and retrieval, we need more benchmarks beyond MTEB that are less vulnerable to overfitting. That’s why RTEB was just beta-launched! ⚖️ Both open and held-out datasets to prevent overfitting to evaluation sets. 🌍 Realistic datasets from critical enterprise domains like law, healthcare, code, and finance. 🔎 Only focus on retrieval applications with relevant large-scale datasets. Check out the blog and leaderboard on @huggingface and join the community in building a stronger, more reliable benchmark. Blog:
View the Leaderboard: https://t.co/gA2LPyngBw
Context
The Retrieval , or RTEB, tests whether search models can find relevant documents beyond familiar public test sets. Its launch explanation describes a combination of open datasets, whose results others can reproduce, and private datasets evaluated by benchmark maintainers. Keeping part of the test private is intended to make training specifically for the test harder.
The useful signal is how performance carries over to less-exposed data. The benchmark focuses on retrieval across domains including law, healthcare, code and finance. Its authors also acknowledge that about half the launch datasets were adapted from question-answering tasks, which can favor keyword overlap. Private improves the evidence available for model selection, while task design still shapes what the score means.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…