Predicting which unevaluated model would top a benchmark, then checking
- Source
- x.com
- Date
Most models are only evaluated on a fraction of the benchmarks out there. ArtifactLinker, our new system, predicts which ones would set a new state-of-the-art on benchmarks hosted on @HuggingFace, then runs the evaluation to verify. 🧵
@huggingface ArtifactLinker is built on a graph of HuggingFace data—models & datasets are nodes, and reported eval scores form the edges. We trained a GNN for it to rank which models are likely to reach a new SOTA on which benchmarks, beating prompting-based LLMs.
@huggingface In ArtifactLinker, an LLM coding agent writes and runs the evaluation code, with shared memory across runs. We found that it comes within 80% of the officially reported score 72.6% of the time.
@huggingface Using ArtifactLinker, we found cases where a strong model had never been evaluated on a benchmark it would set – or near-match – the SOTA on. We also found that newer LLMs like Gemma often lose to older DeBERTa models on natural language inference tasks.
Context
Ai2's ArtifactLinker searches for promising model evaluations that nobody has reported yet. It represents models and datasets as a network connected by existing scores, then learns from those relationships to rank untested model– combinations. A coding subsequently writes and runs the evaluation. The project announcement describes this prediction-then-testing workflow as a way to find strong models overlooked by incomplete benchmark coverage. The ranking chooses what to investigate; running the test supplies the evidence, though the agent's runs are imperfect: Ai2 reports it came within 80% of the officially reported score 72.6% of the time.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleFeaturing Every Eval Ever Results on Hugging Face Model Pageshuggingface.co
articleOpenAI’s accidental cyberattack against Hugging Face is science fiction that happenedSimon Willison
articleEvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language ModelsXinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, Robert Kirk
Checking sign-in…
Loading comments…