Claude Code Ships Plugin Evals to Catch Skill Regressions
- Source
- x.com
- Date
Claude Code added a new `claude plugin eval init` command that lets developers define concrete test cases for a plugin or skill, then run it with and without the plugin installed to score the delta. Skills and plugins have historically degraded silently across model releases, with builders discovering the breakage only after it reached production and users noticed something felt worse. This gives plugin authors a repeatable check to run before every model upgrade, turning a manual guess into a measured comparison. It also gives users of third-party plugins a way to demand evidence that a plugin still helps before they keep it installed. The practical effect is fewer silent regressions reaching production agents, and a faster path to catching them when a new model does change behavior underneath a plugin that used to work.
we heard feedback that it's hard to know if your skills are still working with new model releases plugin evals are here to help run `claude plugin eval init` in your plugin folder https://t.co/Q0I1ZnugDs
Context
Claude Code lets developers extend it with skills and plugins, add-ons that give it extra instructions or capabilities for a specific task. Thariq, who works on Claude Code, said the team heard feedback that it's hard to know whether a plugin still works correctly after a new Claude model release.
In response, Claude Code added plugin evals, run by typing `claude plugin init` inside a plugin's folder. A fuller description of the feature says it lets a developer draft test cases, run the plugin against them, score the results, then rerun the same tests without the plugin to see what it actually changed, reported both in the terminal and as an HTML report.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
Checking sign-in…
Loading comments…