eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Gives you precise, shared terminology for building AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → harnesses and explains specifically why agentic evals fail differently than single-turn evals (compounding errors, valid-but-unexpected solutions), which helps you design graders and suites that don't break on production agents.
Key quotes
“These same capabilities that make AI agents useful—autonomy, intelligence, and flexibility—also make them harder to evaluate.”
“Without them, it’s easy to get stuck in reactive loops—catching issues only in production, where fixing one failure creates others.”
“It “failed” the evaluation as written, but actually came up with a better solution for the user.”
“Agents use tools across many turns, modifying state in the environment and adapting as they go—which means mistakes can propagate and compound.”
