I Made 17 AIs Play 7,700 Hands of Poker

Chrystian Schutz built HoldemBench to answer a practical question: if an AI has to make a sequence of decisions under uncertainty, should it be one of the new decision models, a cheap chat model, an expensive frontier model, a model running on your own GPU, or a plain script? Poker suits the question because every decision has a price in chips and luck can be measured and partly removed. Schutz's write-up covers 17 AIs and 7,700 hands of heads-up no-limit Texas hold'em, and its headline finding is that cheap reasoning models played about as well as the frontier at a fraction of the cost, while the decision models lost to a 30-line Python script.

Because a few hundred hands of raw chip counts mostly measure luck, the design matters. Every deal is played twice with the seats swapped, every pairing in a league uses the same deals, and when both players are all in before the river the hand is scored by each player's equity at that moment rather than by the cards that came, which Schutz says alone cut the spread of results by a factor of 2.7. Players are rated in big blinds won per hand, fitted across all matches. Even so, Schutz estimates that pinning one pairing to within one big blind per hand takes about 320 deals, and their own run shows why: after 20 deals Claude Opus 5.5 led the script by 3.5 big blinds per hand with an interval that excluded zero, and after 50 more the match was a draw.

On that basis, DeepSeek's V4.1 Flash, Muse Spark Contributor and GPT-6 Luna finished level with or above OpenAI's GPT-6.1 Sol and Anthropic's Claude Opus 5.5, and Luna and Muse drew with both head to head. Luna cost $0.71 per thousand hands, Opus $22. The four decision models, endpoints that take a state and a list of options and return a choice with probabilities, were cheap, fast and deterministic but played strange poker. The best, Jev 1.13, lost to the script by 2.7 big blinds per hand, Solar Decide went all in in 56% of its hands, pplx folded hands it was ahead with on the river, and all four underestimated their own chance of winning by 15 to 19 points. The script, which estimates its equity with a quick simulation and bets when ahead, drew with Opus, although Sol beat it. Bielik, a Polish open-weights model run locally, finished last, below a bot that picks actions at random, and never once folded to a bet.

Schutz lists the limits in the write-up. Heads-up play with stacks reset every hand and five fixed bet sizes is a narrow slice of poker, the frontier and local models played fewer deals and so have wider intervals, head-to-head results over 20 deals are hints rather than verdicts, and a different prompt could change the order. None of these players would survive a real poker solver; the point is the comparison between kinds of AI. The code and every decision with its cost are published on GitHub and Hugging Face.