Vals RSI Index: can models build their successors? Muse Wonder Full ranks 9th
Source
vals.ai
Author
Vals AI
Date
Why it matters
Measures whether agents can do open-ended ML research under fixed compute. Leaders beat a published LM training reference but still lag on the others. Profiles show which research habits help, such as verification before promotion.
Key takeaways · AI-distilled
Vals scores four autonomous research tasks, compression, LM training, judge-agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → engineering and post-training, on a log scale where 0 is the starting baseline and 0.5 a published, human or frontier reference; the index averages them.
In LM training Claude Opus 5.5 reached 0.8308 bits per byte, beating the published 0.85 reference with a recipe trained 30 minutes on one H100 per seed, though Vals says its protocol is not a matched reproduction of the original paper.
Despite having an H100, the leading enwik8 compressors used adaptive context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → mixing rather than neural compression, and none reached the 1.2435 BPB nncp reference; Claude Opus 5.5 led at 1.3120.
Experiment volume did not predict quality: GPT-5.6 Sol logged 114 LM-training trials yet trailed most models, while Claude Fable 5.1 ran 47 and placed second; Vals says idea choice and final-candidate checks mattered more.
Only six of 25 agents beat the 35.38% parent in post-training; Gemini 3.8 Flash fell to 3.54% after picking a checkpoint on simulated tool runs, while Claude Fable 5.1 rejected its adapters and gained via prompt guidance alone.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
