• determinism — Whether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
• tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
• human-in-the-loop — Requiring a person's approval at specific points in an automated process, chosen so the irreversible steps are the ones a human sees.

If you're moving agents from demos to production, this lays out concrete architectural patterns for continuous evaluation — evaluating multi-step tool use and planning, detecting drift and unsafe behavior, and building human-in-the-loop feedback loops that static offline benchmarks can't capture.

• videoWhy AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax
• videoYour company brain will leak secrets: how we stopped it for big banks — Tanmai Gopal, PromptQL
• videoTethered: Our Agents Are Us — Shu Fang, Two Sigma
• videoAgents' next frontier: agent-to-agent and network effects — Jean-Denis Greze, Town

• articleDemystifying evals for AI agentsAnthropic Engineering
• postEvaluating agents component by component, not just end to endAndrew Ng
• articleSelecting The Right AI Evals ToolHamel Husain

Checking sign-in…

Loading comments…