⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
The Factorio Learning Environment tests language models in Factorio, a game about building automated factories. On the Latent Space podcast, creators Jack Hopkins and Mart Bakler explain that the model writes Python that calls high-level game actions, and is scored on production rate plus milestones such as items made for the first time. Lab play gives fixed target items and mostly tests spatial reasoning. Open play asks for the biggest factory possible, so the model must set its own goals. Their paper says models still lack strong spatial reasoning.
Among the models of that period, such as Claude 3.5 Sonnet and GPT-4, a creator describes a results chart in which Claude scores almost twice the nearest model, and says Claude beat the others in both settings without a clear reason why. DeepSeek was decent in lab play but poor in open play, at one point building rows of chests. The creators suspect Claude is better trained for long-term planning, which is untested, and results for reasoning models were preliminary and unpublished.
Checking sign-in…
Loading comments…






