How to really stop your agents from making the same mistakes
Source
Garry Tan
Author
Garry TanTop Viber
Date
Key takeaways · AI-distilled
A agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition → is a markdown procedure that teaches the model how to approach a task, not what the answer is. Think method call: the same procedure produces radically different output depending on what the user passes in.
The fix that killed a recurring failure was one hard rule: live calendar APIs are only for the future or the last 48 hours, everything historical goes through the local index first. Five minutes of blocked API retries became a sub-100ms local search.
The missing practice is an ordered sequence, not more tooling: failure happens, write the skill, write the deterministic code, write unit tests, write the model evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, add a routing trigger, audit for duplicates, smoke test, file it correctly.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
Why it matters
A skeptical look at AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → evaluation and observability — using LangSmith's sophistication as a foil — and what actually stops an agent repeating failures versus what merely measures them. Read it if you are drowning in traces but the agent keeps making the same mistake.
Key quotes
“$160 million in funding, and most LangChain users still don't test their agents, because the framework gave them a gym membership without a workout plan.”
“But pieces aren't a practice.”
“The model's intelligence created the constraint that prevents the model from being stupid.”
“That's the bug. Not a wrong answer. A wrong side.”
