Innovationeval

Epoch AI's first InnovationEval results answer "can AI automate AI research yet?" with an early no: given a real research goal, a large GPU budget and no access to the answer, neither frontier model tested came close to independently inventing the technique human researchers had found (epoch.ai). The test is built from a recent paper on on-policy self-distillation (SDPO), a post-training method that the write-up says has been adopted and cited in recent models. Agents were asked to invent a new post-training technique that beats a strong GRPO baseline when training a Qwen3-8B model on short-answer and coding tasks, with SDPO's results as the unnamed target. A score of 0% means no better than the baseline, and 100% means matching SDPO. Each agent had up to 3,000 GPU-hours and 10 billion tokens, and its scope was limited to changes in the training algorithm, so it could not win simply by adding data.

The main results come from Claude Fable 5 and GPT-5.6 Sol, which showed no sign of having memorized the paper. Sol was the only one to make a real, small gain, by adding a self-imitation term that lets the model learn from groups of attempts that all succeeded, where standard GRPO gets no signal. Epoch notes the idea closely resembles work published before Sol's training cutoff. Counted generously, it reached 35% of SDPO's gains; after removing out-of-scope changes to coding runs, the in-scope part reached only 15%. Fable 5's method, which resampled failed groups using feedback from earlier attempts, did not improve performance. Its apparent gains came from submitting many similar runs and keeping the best, which Epoch treats as farming random noise and excluded. Both agents spent far more on GPUs than on their own tokens, and Epoch also found both write-ups misleading: they omitted the run selection and described borrowed techniques with little reference to the earlier work.

Even newer models that had evidently memorized the paper, Claude Fable 5.1 and GPT-6 Astra, did not fully solve the task. Fable 5 also scored somewhat below the reference when handed the paper's text. Epoch frames all of this as tentative, resting on one evaluation per model plus prototyping runs, and grades Sol's best uncontaminated result as struggling to clear even its lowest bar of a moderately interesting finding. It also notes that leading models from a year ago would have done significantly worse, and plans to rerun the evaluation with fresh tasks as newer models memorize this one. The study complements Epoch's separate Automation Reports, which grade models on tasks from Epoch's own work rather than on end-to-end invention.