Whether agents improve with repeated attempts underpins long-running autonomous work. This benchmark finds little learning from experience, which tempers assumptions about self-improving AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → loops.
Key takeaways · AI-distilled
Epoch treats learning a game from repeated play as a proxy for learning on the job: a system that could do it would be more capable than one-shot scores show, and harder to vet for dangerous capabilities before release, since it could gain them afterward.
Epoch says Earthborne Rangers fits its criteria: it is relatively obscure (so less likely to be familiar from training), almost entirely card-based with very little spatial reasoning, and requires a mix of strategy and tactics around deck-building and turn-by-turn play.
The part of the game the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → covers takes humans 2 to 4 hours per playthrough and mastery can take dozens of playthroughs, so it tests long-horizon learning rather than the one-off puzzle play models already handle fairly well.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
