Most AI agent skills are written like tutorials. The data says they work better as runbooks.
A new study analyzed 8,135 agent trials to understand when skills actually help.
The main finding: skills do not mainly fill knowledge gaps.
They stabilize execution.
Procedural anchoring accounted for 65.7% of successful skill cases. Explicit knowledge injection accounted for just 4.5%.
That changes how you should build a skill library.
> Distill successful trajectories into step-by-step procedures
> Label failed runs before using them for skill generation
> Route by domain before retrieving from large catalogs
> Use strict triggers when skills look semantically similar
One result shows why this matters.
When success and failure labels were visible during skill distillation, an agent reached 74.6% success.
Remove those labels, and it dropped to 40%.
The retrieval results were even stranger.
As the catalog grew from 5 to 100 skills, retrieval precision collapsed. Task success barely moved.
Agents often picked the wrong skill and still extracted enough useful procedure to finish the task.
So the important question is not simply:
“Did the agent retrieve the correct skill?”
So the important question is not simply:
“Did the agent retrieve the correct skill?”
It is:
“Did the retrieved skill give the agent the right procedure at the right moment?”
Full breakdown of what 8,100+ trials reveal about how to design, distill, and retrieve agent skills:
https://t.co/dokcBZJXdS
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → skills work by anchoring procedure, not filling knowledge gaps — so distill runbooks from success/failure-labeled trajectories and route by domain before retrieving from a large catalog.
