Vibeleaderboard
← All Intel
Intel / video

Benchmarking LLMs at the Game Of Science (Eleusis)

Source
youtube.com
Author
Hugging Face
Date

Context

A Hugging Face video presents a that tests whether language models can do the loop at the heart of science: form a hypothesis, run an experiment, revise. It uses Eleusis, a card game in which a hidden rule decides which cards are accepted. A model plays cards one at a time, gets accept or reject feedback, states its current guess and a confidence, and decides when to commit. Each turn costs a point and each wrong guess costs two more, so guessing early is risky and waiting is costly.

The presenters report that Gemini 3 Pro from Google DeepMind and Claude Opus 4.5 from Anthropic led, scoring about 17, with the Kimi K2 Thinking close behind. They also separate finding the rule from knowing when to commit, and say GPT-5.2 rarely guessed wrong but held the correct rule for over four turns on average before committing. The results are the authors' own, and only the spoken figures are used here.

Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
Read the source www.youtube.com
More from Hugging Face
Recommended reads
Comments

Checking sign-in…

Loading comments…