Agentic Evaluations Workshop - Deep Dive on the Future on Evals for Agents.

This Hugging Face workshop brings together five talks on evaluating AI agents, and the thread running through them is that a pass rate stops describing an agent once the system acts over many steps. Avijit Ghosh, a policy researcher at Hugging Face who co-leads the EvalEval coalition, opens with two agents that both pass the same task, though one takes fewer steps, costs less, finishes sooner and never errors. He adds that a model name does not identify an agent: he wants reports to record the sub-agents, MCP servers, memory configuration and per-session data, so that a third party can reproduce the run.

Arvind Narayanan of Princeton gives the sharpest version of the argument. His team's paper, Towards a Science of AI Agent Reliability, treats reliability as separate from capability. It breaks reliability into consistency (does the agent get the same result, by the same actions, when it repeats a task), robustness (does it cope with injected faults such as API timeouts, or with a reworded prompt), predictability (does its stated confidence separate its successes from its failures) and how severe its failures are. A 70 percent score, he points out, can mean the agent reliably solves 70 percent of tasks, or that any given task works 70 percent of the time. On Gaia and the Tau-bench airline benchmark, he says, accuracy across frontier models rose quickly over 18 months while the combined reliability score rose only gradually. He calls these findings tentative, based on two benchmarks so far, and argues that the gap matters most for agents meant to work without a person checking their output.

Pierre, a research engineer at Meta, describes Gaia 2, which tests agents in a world that keeps changing. It runs on Meta's Agents Research Environments, which simulates a set of phone-style apps such as email, calendar and messaging. New messages and cancellations arrive while the agent works, and simulated time can jump forward. Grading compares the actions the agent took with an annotated expected sequence, and an LLM is used only to judge free text such as an email's wording. In a results table from last year, which he called outdated, models did well on search and execution tasks but scored around zero on tasks that depend on time passing.

The last two talks turn to practice and reporting. Mahesh of Bespoke Labs urges teams to build a sandboxed environment, tasks and grader before trying to improve an agent. He also tracks steps, tokens, latency and cost alongside success rate, and notes that the same model scores differently in different harnesses. Nathan Habib presents Hugging Face Community Evals, where a benchmark dataset defines how it is run and anyone can submit a score as a pull request on a model's repository, which then appears on that benchmark's leaderboard for others to dispute. A later post shows Every Eval Ever results appearing on model pages through Community Evals, which joins Ghosh's reporting format to Nathan's leaderboards.