Vending-Bench 2
andonlabs.com- Category
- Developer Tools
- Rank
- No. 2347Tools index
Previous survey · No. 2211 ·
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Date
About
Tests whether an agent can run a simulated vending machine business profitably and stay coherent over a full simulated year. Compare the run length and tool setup before comparing final balances.
What it does
Andon Labs describes Vending-Bench 2 as a benchmark in which an AI agent runs a simulated vending machine business for a year. The page says models are ranked by the money left in their bank account at the end of the run.
Stated on the product site
- Scoring
- Money balance after one simulated year
- Starting conditions
- A $500 starting balance and a $2 daily machine fee
- Early termination
- A run ends if the agent fails to pay the daily fee for more than 10 consecutive days
- Simulated suppliers
- Suppliers may be adversarial, deliveries can be delayed, and customers can demand refunds
- Multi-agent variant
- Vending-Bench Arena puts competing agents at the same location
- Run scale
- The page reports 3000 to 6000 messages per full-year run
Not stated on the site
- How many runs sit behind each leaderboard average, and are the error bars standard deviations or standard errors?
- Is the simulation environment available for others to run, or only the published results?
- How was the estimate of roughly $63k a year for a good strategy checked against human play?
Written from the product site at andonlabs.com.
Tags
benchmarkevaluation
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.