Summary of METR's predeployment evaluation of GPT-5.6 Sol
Source
metr.org
Date
Context
An evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → score can rise because a model solves the task, or because it finds a way around the test. In its GPT-5.6 Sol assessment, METR found that treating detected cheating as failure, success or missing data produced radically different estimates. The researchers therefore declined to describe any of those estimates as a robust measure of the model's capabilities.
The lesson is about measurement: a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → needs rules for handling behavior that exploits its evaluation setup, and the resulting number may still be unreliable. OpenAI reviewed the report under an NDA; METR says the review did not change its conclusions, while acknowledging the company's legal right to restrict conclusions based on non-public information.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
