Popular Claude Code skills vs. a placebo: 2 beat it, 1 did worse

Most popular skills did no better than same-length neutral text, and none cut cost. Engineers can test whether a skill earns its context before adopting it.

• SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
• agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

• articleLinting 216 public Claude Code skills – 69% won't reliably triggersgharlow
• articleDesigning AI-resistant technical evaluationsAnthropic Engineering
• articleWe removed ~80% of the Claude Code system prompt — what we learned about writing system prompts, skills, and CLAUDE.mdsThariq

Checking sign-in…

Loading comments…