← All IntelClip / EducationClaude ships an eval framework for skills
From Full Walkthrough: Writing & Using Skills — Nick Nisi and Zack Proser · ≈58:20
There is now a built-in way to check whether a skill helps or hurts, which removes the main excuse for shipping unmeasured prompt artifacts.
What’s in it
- There is now a built-in way to check whether a skill helps or hurts, which removes the main excuse for shipping unmeasured prompt artifacts.
Clip transcript
Do you want to talk about um any of these topics, Zach? I can talk about when confidence when um yeah, confidence scoring saved us. >> Yeah, what's that? >> That was when we were working on the um when confidence scoring saved us. The uh well, that was kind of built into the evals uh that we wrote. Like Claude ships with a whole eval framework now that you can use and it'll like spin up a GUI for you or like a it'll create an HTML report and you can see like before and after and all of this insight into how your skills are running and whether they're actually like improving Claude or making Claude worse at the task. Um but before that existed, uh I was writing my own to do that and uh it was all based on on that. And so like let me let me bring up the um
Comments
Checking sign-in…
Loading comments…