ArtifactLinker predicts untested SOTA benchmark results, then verifies them

ArtifactLinker is a research system from Ai2 that looks for model and benchmark pairings nobody has tested and checks whether any would set a new best score. The problem is coverage: models published on Hugging Face usually report results on only a fraction of the datasets they could be run on, so the strongest model for a task can go unnoticed. The paper treats Hugging Face as a network in which models and datasets are points and each reported evaluation score is a line between a model and a dataset. Finding an unmeasured result then becomes predicting which lines are missing and what their scores would be.

The system works in two stages. First, a ranking model learns from the existing connections, drawn from Hugging Face models and datasets with more than 1,000 monthly downloads, and scores the untested pairs. In the paper's tests, graph neural networks, which learn from the pattern of connections around each model and dataset, ranked missing links better than language-model rerankers, though language models did better at predicting the score itself when a dataset had few recorded evaluations. Second, an AI coding agent takes the top candidates, writes evaluation code and runs it. The ranking only chooses what to test; the agent's run is what produces the evidence.

That verification step is imperfect. To measure it, the authors had the agent rerun 263 evaluations that already had official scores and counted a success when it reached at least 80% of the reported score. It succeeded 72.6% of the time. In a case study on natural language inference, judging whether one sentence follows from or contradicts another, the system examined 45 models on 12 datasets. For one DeBERTa-based multi-task model whose model card listed other NLI results but not SNLI, it produced what the authors believe is the first reported SNLI score for that model, 92.1% accuracy, within about a point of the leaderboard best of 93.1%. The system also found that newer models including Google's Gemma family often trailed older DeBERTa models on these tasks, a pattern the authors note also appears in earlier research. The code and the benchmark dataset built from the network are public.