GardenBench: A lightweight, daily evaluation of LLM capabilities

GardenBench is built to catch something most benchmarks miss: a model behind the same name can score differently from one day to the next, including after its maker changes it. The authors run a fixed set of small tests against hosted models on a schedule, usually daily, and log accuracy, cost and response time over time. The 11 suites mix image tasks, such as identifying birds, books or countries and reading graphs or transit maps, with text tasks such as ciphers, puzzles and finding a fact buried in long text. Answers are scored by exact match with no partial credit, web search is off, and the test items are kept unpublished to limit contamination.

In the two weeks analysed, 16 models from seven providers, the authors found rankings that flipped between days but no steady rise or fall for any model. GPT-5.6 Sol averaged 73% against GPT-5.5's 72%, yet GPT-5.5 came out ahead on two separate days. In an earlier trial run, GPT-5.6 Sol hit an all-time high of almost 75% on July 26 and then declined, and the paper notes that on July 29 OpenAI said the model had been altered after deployment. DeepSeek's V4 Flash and V4 Pro fluctuated most.

The trade-off is depth. About 100 tests in total is cheap, roughly $50 for the complete benchmark by the authors' estimate, and each full run takes up to an hour, but small suites and strict exact-match scoring make individual daily scores noisy, and a correct answer wrapped in extra text counts as wrong. The paper itself gives slightly different test counts in different sections, and the authors note that the text-only models were tested on only four suites and may look stronger than they are. The two-week window was chosen to show day-to-day swings rather than long-term trends. The paper is a preprint, and results are posted to a public dashboard.