For these tasks, it checks the resulting system state after each turn, such as whether a requested file exists with the correct contents. It also checks required calls because reading information may leave that state unchanged. Together, these checks allow some variation in the route an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → takes while still testing whether it performed the necessary work.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoWorld Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
videoExtreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
videoInside Abridge: The AI Listening to 100 Million Doctor Visits — Abridge's Janie Lee & Chai Asawa
videoExo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research
