Introducing Owe Bench

Mercor, which recruits domain experts to build AI training and evaluation work, has released OWE-Bench, 26 engineering and research tasks that each give an AI agent 4 to 18 hours (median 8) to make real progress on a project. In some tasks the agent starts from a working codebase and has to push a measurable target, such as a GPU kernel's speed or a model's error. In others it builds the whole thing from an empty environment: a trained model, a data pipeline, a mechanical design. The headline number needs one piece of context. Every task reports results in its own units, so Mercor rescales them: the untouched starting point scores 0% and the task's ceiling scores 100%, and an agent that gets halfway there earns 50% instead of a failing zero. A model's score on a task is the average of eight attempts, and its leaderboard score is the average across all 26 tasks. So when Anthropic's Claude Fable 5.2 tops the board at 54.6%, it means its work landed on average a little past the midpoint between starting point and ceiling, not that it solved 55% of the tasks. OpenAI's GPT-6 Astra follows at 44.2% and Claude Opus 5.5 at 41.6%, and Mercor says no model is close to the ceiling. That partial credit differs from Mercor's earlier Physical-Engineering's Last Exam, which awards points for cost efficiency only after a design clears every hard safety and feasibility check.

The grading is built so an agent cannot score by convincing itself, or a judge, that its work is good. The agent hands over a finished artifact, such as a model checkpoint, a kernel package or an aircraft design, and an independent grader tests it offline, with no network access, on held-out inputs the agent never saw. It measures by running, simulating or geometrically checking the artifact rather than asking a model's opinion. The aircraft task, for example, is scored over 34 hidden flights, and a chess post-training task on 1,000 private puzzles. Mercor also looked for reward hacking, where an agent games the scoring instead of doing the work: it audited all 2,912 trials, found exploit attempts in 39 (1.3%), and set to zero any exploit that raised a score. Hidden tests also explain the main way agents failed. Mercor read 624 runs from Claude Opus 5.5, Google's Gemini 4 Argon and GPT-6.1 Sol and found that most failed runs checked their code on easier cases than the task required, such as small inputs, familiar data or the agent's own simulator, then shipped work that broke at the real scale, temperature or input shape. GPT-6.1 Sol and Gemini 4 Argon also lost points by overfitting to practice data, for example adding shortcuts that only work on the public samples.

These are Mercor's own results on a benchmark it built, and 26 tasks is a small base, so the per-domain comparisons carry less weight than the overall ranking. Mercor reports Fable 5.2 leading every domain, but by very different margins: 62.2% against 49.7% for Claude Opus 5.5 in software engineering, and only 59.6% against 57.3% in physical engineering and 40.2% against 39.4% in scientific computing, where Mercor says GPT-6 Astra is within the margin of error. Price did not track score either: Astra spent $234.63 on tokens per attempt for 44.2%, Fable 5.2 spent $105.08 for 54.6%, and Claude Sonnet 5.5 spent $38.65 for 39.0%. Mercor has published every task, including the hidden test sets and the same graders used for the leaderboard, so outside groups can rerun the numbers rather than take them on trust. Until someone does, the fair reading is that the strongest current agents get roughly halfway on multi-hour engineering projects by Mercor's measure, and that checking their own work against realistic conditions is where they most often fall short.