A model can know the rules of a game and still lose track of where it is. ARC Prize designed SnakeBench to test that gap: models receive a text representation of a changing board and choose their next move while avoiding other snakes. The organizers found that many models collided early despite receiving explicit coordinates and rules.
The experiment also exposed a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →-design problem. Supplying more information or preventing illegal moves could improve play, but those interventions would contribute the developer's reasoning to the result. SnakeBench was an exploratory side project informing future benchmark design; its rankings describe performance in this particular game and prompting setup.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
