Cua's test of agents in KiCad, an electronics-design application, shows how partial credit can change a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →'s leader. Cua ran 25 tasks under a 200-step limit, using early access to Google's Computer Use tool with Gemini 3.5 Flash. Gemini completed five tasks and partially completed three. OpenAI's GPT-5.5 completed six and received no partial credit. The reported results gave Gemini the higher mean reward per task, 0.267 versus 0.240.
The two measures answer different questions: completed-task counts capture finished work, while the average score also rewards incomplete progress. Gemini led the latter measure in this evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →; GPT-5.5 delivered more fully completed tasks. Neither finished more than six of the 25.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
