Vibeleaderboard
← All Intel
Intel / article

An Empirical Study of Agent Skills' Downstream Utility

Source
arxiv.org
Author
Yu Cheng, Dehai Zhao, Zhongxin Liu, Qing Huang, Zhenchang Xing, Xiaoxue Ren
Date
From the Daily Brief

The researchers ran the same tasks with and without Skills across nine configurations to isolate what a Skill adds. Installing a relevant Skill did not reliably raise pass rates. On more than a third of tasks, a Skill that helped in one configuration hurt in another, so the benefit belongs to the pairing of Skill, model and harness rather than to the Skill itself. Execution traces point to a mechanism: the procedures a Skill prescribes can become an execution burden, with the agent spending effort following steps it would not otherwise need. This adds to earlier placebo testing of popular Claude Code skills, which found few beat neutral text. If you publish or depend on Skills, test them against a baseline in each model and harness you actually run instead of assuming they transfer.

Read the 2026-10-09 Brief →

Context

A relevant does not reliably help an AI agent, according to a study. Skills are reusable packages of procedural guidance and resources. The authors define a Skill's utility as the change in pass rate versus the same tasks with no Skill, under the same model and harness (the software that runs the model as an agent), and measure it on 87 SkillsBench tasks across nine configurations.

The same Skills helped some configurations and hurt others on 36.78% of tasks, and execution traces showed recommended procedures sometimes becoming a burden. Ranking candidate Skills by relevance overlooked more useful ones: within the evaluated candidate sets, by support for the operations a task requires raised first-choice pass rates by 4.35 to 5.80 percentage points across three configurations.

The reranking gain was measured only within those candidate sets, so it is not a general ranking method.

Key takeaways · AI-distilled
  • The study measured utility as the pass-rate difference from a No-Skill run on 87 SkillsBench tasks; the same Skills helped some of nine model-harness configurations and hurt others on 36.78% of tasks.
  • Relevance rankings missed more useful candidates: reranking marketplace Skills by support for the task's required operations raised first-choice pass rates by 4.35 to 5.80 points across three configurations.
  • When several Skills are supplied, organizing them as a Stage Plan or DAG beat use order alone, with the DAG's extra gains concentrated in tasks given five or six Skills.
  • The authors derive 17 authoring practices linking executable procedures to recovery, preserving task requirements, and checks on final artifacts, based on -assisted trace analysis followed by author review.
Terms in this piece · Glossary
  • agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
  • reranking — A second pass that re-scores retrieved candidates by reading each one against the query, fixing the ordering that fast vector search got approximately right.
  • Dependency — Anything a program needs to run that its author did not write: the language, its standard library, or a separately installed package.
  • Large language model — A model trained on large amounts of data to process and generate language.
Recommended reads
Comments

Checking sign-in…

Loading comments…