Vibeleaderboard
← All Intel
Intel / repo

Popular Claude Code skills vs. a placebo: 2 beat it, 1 did worse

Source
github.com
Author
simonether
Date
From the Daily Brief

Skills are cheap to install, and popularity is often the only signal developers use to pick them. This open experiment adds a control. Each of nine top-starred Claude Code skills was compared with a neutral placebo of the same length, across tasks from SWE-bench Verified, Terminal-Bench and TBLite. The placebo matters because any extra context can change agent behavior, so a skill has to beat text that merely takes up the same space. Only two skills beat the placebo. One performed worse, and none reduced cost. The result covers nine skills and three benchmarks, so it does not settle the value of skills in general. It does give a method any team can reuse: before adopting a skill, run your own tasks with the skill, without it, and with a same-length placebo.

Read the 2026-10-07 Brief →

Context

A GitHub project called skill-placebo tests whether popular Claude Code skills, bundles of instructions an loads, do better than neutral filler text of the same length. Each of nine skills was compared with no and with a placebo sized to match its always-loaded footprint, on 15 public coding tasks with claude-opus-5-5 in Claude Code, 30 trials per arm.

The author reports that two skills beat the placebo on cost: ponytail, about 12% cheaper, and agent-skills, about 5% cheaper. One, planning-with-files, did worse, with an 80% pass rate against the placebo's 100%, after an amendment that counted two timed-out runs as failures. The other six were no better, and no skill was measurably cheaper than using none.

The author lists limits: one model, public tasks likely seen in training, most tasks already at the model's ceiling so the result rests mainly on cost, and pass-rate intervals of about 10 points. This is a first version, so it shows how a placebo control works more than it ranks which skills are worth installing.

Terms in this piece · Glossary
  • agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Recommended reads
Comments

Checking sign-in…

Loading comments…