The Agent Said It Was Done. The Database Disagreed.
- Source
- huggingface.co
- Date
Most agent benchmarks check whether the tool calls were well formed or whether the final answer matched. ThinkingBox checks the thing users actually depend on: the state of the database after the agent says it is done. Microsoft built 507 stateful workflows over MCP tools and runs each one 20 times, so a model that succeeds once but drifts on repeats is scored as unreliable rather than capable. The repetition also exposes the cost of consistency, since running the same task many times is how teams discover flaky behavior in practice. The benchmark ships through OpenEnv, which makes it usable outside Microsoft. If your agents write to real systems, the lesson transfers directly: grade the terminal state of the records, and run each evaluation case more than once before trusting a pass.
Context
Microsoft's ThinkingBox evaluates whether an leaves business records in the required state. In its published example, an agent correctly consults a refund policy but closes a delivery-support ticket that should remain on hold. Valid tool calls and a confident completion message conceal the wrong outcome.
The checks final records and unintended changes across 507 workflows, repeating each task 20 times from a clean starting state. That distinguishes occasionally completing a task from completing it consistently. For systems that modify business records, the resulting database state supplies evidence that the agent's own account cannot.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
Checking sign-in…
Loading comments…