Agents silently guess past missing requirements instead of asking. Shows model-specific failure modes (confident wrong beliefs, detecting but not acting, vague questions) and gives a reproducible agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → to test your own agents.
Key takeaways · AI-distilled
Scale's HiL-Bench hides missing, ambiguous or contradictory details in SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → Pro and BIRD tasks. Frontier agents that solve 75% to 89% with full information drop as low as 4%, filling gaps with confident assumptions.
Giving agents a clarification tool did not fix this when they chose when to use it: performance peaked at 38% on SQL and 12% on software tasks, so the mechanism to ask existed without the judgment to use it.
Across more than 3,600 failure traces, GPT models built on wrong beliefs without noticing gaps, Claude recognized it was stuck in 45% of its alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition → failures yet submitted anyway, and Gemini asked often but too vaguely.
Reinforcement learning on a 32B model raised Ask-F1 by 28 points on SQL and 17 on software tasks, and a model trained only on SQL also asked better on unseen software tasks, which the authors read as a transferable agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition →.
Terms in this piece · Glossary
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
