← All IntelClip / EducationHow they actually evaluate a skill
From Full Walkthrough: Writing & Using Skills — Nick Nisi and Zack Proser · ≈29:31
A concrete, reproducible eval design — run the task with and without the skill, grade against a rubric, fail on regression, and re-run as new models drop.
What’s in it
- A concrete, reproducible eval design — run the task with and without the skill, grade against a rubric, fail on regression, and re-run as new models drop.
Clip transcript
>> Yeah. Great questions. Um yes, sir. Sorry. >> I forgot what I was going to ask. Yeah. Uh so it's actually two questions. The first one is do you do any like formal skill evaluations like a skill benchmark so that as you model drops, which skills are relevant? >> Yes. Um in the the public skills specifically on the the ones that I use internally, I am a little less formal about it. Uh but the ones that we actually ship, uh we do ship uh in the Where is it? We have like a whole evals framework uh that we wrote to make sure that it it lives up to the standards that we have. And we're going to we're going to talk about this a little bit, but like it's mostly uh like doing several runs where it will load Claude without the skill and ask it to do a task and then load it with a skill and then it kind of has like a rubric on confidence or or like a a grade that it gives it and it's it it'll fail if that grade is less with the skill than it was without. Uh it also fail like you know, it it tries to be I think 80% above or higher. So like 80% of the time uh or maybe it's 90%. Uh it's going to get this right with the skill and sometimes it gets it right without the skill. So the skill is maybe only adding 1 or 2% to it, but that's something that we track and keep on top of as new models drop. >> Yeah. >> Okay. Yeah. >> It's it's sort of fuzzy math, but it's almost like by having this this established baseline, you can at least test that way. >> Yeah.
Comments
Sign in to comment.
Loading comments…