Whether agents improve with repeated attempts underpins long-running autonomous work. This benchmark finds little learning from experience, which tempers assumptions about self-improving AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → loops.
Key takeaways · AI-distilled
Epoch treats learning a game from repeated play as a proxy for learning on the job: a system that could do it would be more capable than one-shot scores show, and harder to vet for dangerous capabilities before release, since it could gain them afterward.
Earthborne Rangers was chosen because it is obscure enough to be unfamiliar from training, card-based with little spatial reasoning, playable as text so weak multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition → reasoning is ruled out, and mixes deck-building strategy with turn-by-turn tactics.
The part of the game the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → covers takes humans 2 to 4 hours per playthrough and mastery can take dozens of playthroughs, so it tests long-horizon learning rather than the one-off puzzle play models already handle fairly well.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
