CoffeeBench: a 90-day multi-agent supply chain exposes idle drift

Sakana AI and KPMG AZSA, a Japanese audit firm, built CoffeeBench to test whether a language-model agent can keep a business running for 90 simulated days while trading with other businesses that are also run by agents. Earlier tests of this kind, such as Andon Labs' Vending-Bench, had one agent run a vending machine, or several agents run competing machines (background on that benchmark). The CoffeeBench paper instead simulates a coffee supply chain with two farmers, two roasters and two retailers. Each firm is an agent that can message the others, post listings, make and accept offers, pay invoices and use a role tool to grow beans, roast them or set shelf prices. Every action uses 30 minutes of a business day that runs from 9:00 to 19:00, deliveries take a day, invoices fall due after 30 days, stock spoils, and every firm pays a fixed daily operating cost, so an agent that does nothing steadily loses money.

The tested model runs one roaster while Claude Sonnet 4.6 runs the other five firms, and each model gets three runs scored on net income. Every model beat a roaster that never acts, which finished at minus $2,765. OpenAI's GPT-5.5 averaged $3,109 and Anthropic's Claude Opus 4.7 $2,782, and the paper finds that the stronger models spent their actions on making and accepting offers and on messaging farmers and retailers. GPT-5.5 sent about 140 messages per run. Activity alone did not decide the result: Moonshot AI's Kimi K2.6 made nearly as many tool calls as GPT-5.5 but earned $454, and it sold commodity coffee at lower prices than Gemini 3.1 Pro, which had similar revenue and inventory and earned more. Results varied widely between runs, and the authors note that with three runs per model, small differences in the ranking may not be statistically significant.

The clearest finding is a failure that a profit total would not reveal on its own. Claude Haiku 4.5, the only model to lose money on average (minus $630), stopped doing business partway through each of its three runs, on days 26, 66 and 25, and from then on only called the tool that skips to the next day, averaging about 40 idle days out of 90. Its reasoning stayed coherent: on day 26 of one run it wrote that the business was "operating at peak efficiency" and that the remaining 64 days would focus on executing its strategy, then it neither answered messages nor traded. The authors call this idle drift. They say its cause is unclear and offer accumulated long context, or overly cautious action choice possibly driven by implicit concern about its token budget, as candidate explanations. Sakana's announcement says the pattern appeared in all three Haiku runs and in no other model. For anyone who judges an agent by reading its reasoning, these runs show that a sound plan in the log is not evidence the agent is carrying it out.

The benchmark also leaves a lot of room above current models. Under deliberately optimistic assumptions, the authors estimate a roaster could earn about $23,800, so GPT-5.5's average is roughly 13% of that figure. KPMG AZSA's interest is audit: its press release presents CoffeeBench as the first result of joint research on how agents that set prices, negotiate and make payment decisions could affect financial figures and internal controls. An exploratory test pointed that way. Told to hit a $50,000 revenue target or be replaced, agents did not turn to circular trading, a fraud in which firms pass goods around to inflate sales, even though the environment allows it. The authors read this as preliminary evidence that current agents lack the long-horizon coordination such schemes require.