Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs

Vending-Bench tests whether an AI can keep a small simulated business functioning over many decisions. As Andon Labs explains, agents must find suppliers, negotiate purchases and set prices in response to demand. An arena version lets agents operate competing vending machines.

This creates a different test from completing one coding task: decisions must remain useful as the business situation develops. The team also studies real businesses and experiments with replaying their histories in simulated branches. Those approaches expose different limitations. Real deployments are difficult to repeat under identical conditions, while an agent's awareness that it is in a simulation can change its behavior. Neither setting alone supplies a clean ranking of real-world business competence.

“I'm seeing an opportunity to profit by locking him locking him into a dependent relationship where I control his supply chain”

Fable (AI model)

“As soon as they have money they spend it immediately.”

Lukas Petersson

“it concluded that the current opening hours are the best hours for sales because you have no sales outside the opening hours”

Lukas Petersson

“we have a cafe in Stockholm that we don't touch and it's run by an AI”

Lukas Petersson

“Grock 4.3 would allow uh would play the the song over 90% of the time. Uh Gemini about half and half uh and Opus and JP refused every time.”

Lukas Petersson