Introducing the Multimodal Knowledge Agent benchmark

Google DeepMind's new Multimodal Knowledge Agent benchmark, built with the independent evaluator Artificial Analysis, tests whether an AI agent can do professional work that depends on looking at files rather than reading their text. Each of its 200 tasks, written by working experts across 10 sectors and 83 occupations, hands the agent between 2 and 45 files, such as floor plans, circuit schematics, radiographs and survey maps, and asks for a finished deliverable; 88% of tasks require a file such as a spreadsheet, slide deck or report. The technical report argues that existing workplace benchmarks miss this: when Google removed Gemini 4 Carbon's image-viewing tool, its score on OpenAI's GDPval benchmark did not change (88.1% against 88.2%), because the information those tasks need can be pulled out as text. On MKA, the same removal cost Carbon about six points on the headline score, and on photographs and medical images its code-based workarounds failed outright.

Every model runs in the same open-source agent harness, Artificial Analysis's Stirrup, with the same four tools (run code, view an image, search the web, finish), the same sandbox and a limit of 250 turns. A second agent then grades the output against an expert rubric of about 70 yes-or-no checks per task, rendering pages and opening spreadsheet formulas to do so. The grader runs on OpenAI's GPT-6 Luna, a model from outside Google, and in a 50-task study it agreed with human experts on 96.2% of judgments. The headline score, which the report calls Climb-WSR, counts only the rubric checks that a panel of four earlier models did not already pass reliably, so routine formatting checks do not pad the result. Artificial Analysis keeps the 200 tasks private to stop them leaking into training data; 10 separate sample tasks are public on Hugging Face.

Google's post says Carbon places second at a sixth of the leader's cost without naming the leader. The report does: it is Anthropic's Claude Fable 5.2, at 56.6% on Climb-WSR against Carbon's 54.3%, followed by Claude Opus 5.5 at 53.0% and OpenAI's GPT-6 Astra at 50.2%. The authors say the top three are statistically indistinguishable, while Carbon and Fable 5.2 both score clearly above Astra, and the Artificial Analysis leaderboard shows the same top three order. The cost gap comes from how much work each model does. Fable 5.2 averaged 40.5 million tokens, 129 turns and $35.28 per task; Carbon used 7.1 million tokens over 47 turns for $5.63. Astra used the fewest tokens but still cost $6.03 per task because of its higher per-token prices. On this benchmark, then, Carbon matches the best result within the margin of error for far less money and time.

That conclusion has limits. Google designed the benchmark and wrote the report, and the report itself notes several more. Each model ran once per task with a single grading pass, so the error margins cover the choice of tasks but not run-to-run variation. The vision test was run on Carbon only, two of the four earlier models that define the headline score are Gemini models, and costs use list prices at the time of writing.