How to really stop your agents from making the same mistakes
- Source
- x.com
- Author
- Garry TanTop Viber
- Date
Context
Garry Tan's account of recurring agent mistakes identifies a failure that adding another tool does not solve: an can already possess the right tool and still fail to use it. He describes a meeting-time error made despite an existing script that calculated the correct interval.
His proposed repair combines a written procedure with tests of whether the agent selects and runs that procedure. Testing the script checks its calculation; testing the routing checks whether a user's request reaches it. This extends his argument for testing complete AI systems into a specific maintenance practice: turn an observed failure into both a reusable process and a regression test.
- A is a markdown procedure that teaches the model how to approach a task, not what the answer is. Think method call: the same procedure produces radically different output depending on what the user passes in.
- The fix that killed a recurring failure was one hard rule: live calendar APIs are only for the future or the last 48 hours, everything historical goes through the local index first. Five minutes of blocked API retries became a sub-100ms local search.
- The missing practice is an ordered sequence, not more tooling: failure happens, write the skill, write the deterministic code, write unit tests, write the model , add a routing trigger, audit for duplicates, smoke test, file it correctly.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
“$160 million in funding, and most LangChain users still don't test their agents, because the framework gave them a gym membership without a workout plan.”
“But pieces aren't a practice.”
“The model's intelligence created the constraint that prevents the model from being stupid.”
“That's the bug. Not a wrong answer. A wrong side.”
Checking sign-in…
Loading comments…