Quantifies how often frontier models fail everyday shopping, food, gaming and DIY tasks, including hallucinated claims and broken links, and introduces hurdle criteria to limit reward hacking.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
