What 8,135 agent trials say about writing skills as runbooks

Most AI agent skills are written like tutorials. The data says they work better as runbooks. A new study analyzed 8,135 agent trials to understand when skills actually help. The main finding: skills do not mainly fill knowledge gaps. They stabilize execution. Procedural anchoring accounted for 65.7% of successful skill cases. Explicit knowledge injection accounted for just 4.5%. That changes how you should build a skill library. > Distill successful trajectories into step-by-step procedures > Label failed runs before using them for skill generation > Route by domain before retrieving from large catalogs > Use strict triggers when skills look semantically similar One result shows why this matters. When success and failure labels were visible during skill distillation, an agent reached 74.6% success. Remove those labels, and it dropped to 40%. The retrieval results were even stranger. As the catalog grew from 5 to 100 skills, retrieval precision collapsed. Task success barely moved. Agents often picked the wrong skill and still extracted enough useful procedure to finish the task. So the important question is not simply: “Did the agent retrieve the correct skill?”

So the important question is not simply: “Did the agent retrieve the correct skill?” It is: “Did the retrieved skill give the agent the right procedure at the right moment?” Full breakdown of what 8,100+ trials reveal about how to design, distill, and retrieve agent skills: https://t.co/dokcBZJXdS