The Run Receipt Audit: Diagnosing Harness Failures in AI Agents
- Source
- youtube.com
- Author
- AI Engineer
- Date
Context
An can tell a user that something succeeded while failing to preserve the state needed for its next turn. The Run Receipt Audit illustrates this with a delivered reply missing from the durable conversation record. The next response can sound coherent even though the system has lost part of its history.
The proposed remedy is a run receipt: evidence connecting the triggering request, inherited state, permission to act, attempted operation and externally confirmed result. The surrounding software, often called the , must record these transitions. A message appearing on screen proves delivery; a stored, recoverable record establishes what the agent can remember later. Checking both exposes failures that answer-quality tests can miss.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
“The user saw the reply. The system forgot it happened.”
Ben
“Last writer wins is not a consistency model.”
Ben
“Bound the work before the work bounds you.”
Ben
“Own the state, order the mutation and prove the action.”
Ben
videoHow to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
videoBuild Your Own Team of Managed Agents - Jack Wotherspoon, Google DeepMind
videoHow to Build an AI Code Review Agent - Daksh Gupta, Greptile
videoHow to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Checking sign-in…
Loading comments…