"CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies" will be presented at the Failure Modes in Agentic AI Workshop on July 10 at #ICML2026
Paper: https://t.co/qKk1VOBWP0
We introduce CoffeeBench, a benchmark for evaluating how much net income an LLM agent can generate as a coffee roaster over 90 days in a multi-agent economy with two farmers, two roasters, and two retailers. We observe diverse behaviors across models, including an idle-drift failure mode in Claude Haiku 4.5, resulting in substantial differences in business performance.
https://t.co/gOzXx22jU5
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Long-horizon multi-agentUsing several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.Full definition → economies surface failures short evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → cannot, including idle drift where a model reasons about its situation and never trades. The paper and setup are public ahead of an ICML 2026 workshop.
