Clip transcript
>> Uh yeah, and so then this gets back to just kind of in practice the way this works or at least how it has for me and what we kind of recommend is, you know, build an initial skill. Maybe you're doing that yourself in markdown. Maybe you're using the skill builder in Claude and saying I need a skill to do X. You're doing it you're using that skill for a couple of iterations, maybe a couple days, maybe a week. Um you look at what it produces and then, you know, keep in mind that as you're having multiple conversations with say Claude over the course of a week, all of those conversations are even getting saved locally to some degree in JSONL files. And so you can um be honest with the evaluation phase about is this actually improving things? Is it not? Where does the skill fall short? What are the edge cases it's not currently capturing? What's the annoying thing that I've now discovered that I've been running it 7 days that it's missing? And then you kind of iterate. And but again, you're still going faster because you come back to a state that's already working and you say, "These three edge cases are driving me nuts and you also need to be be to like review your own PR's in the future, right? And so then once that loop is is done, you have a skill that's significantly more powerful and then you can keep keep on running from there. But it's kind of like they're sort of evolving over time. Um so they're again like I think of them as like organizational units of where to put kind of, you know, work intelligence. And then over time if you're if you're doing it right, they're getting better. >> Yeah. When when skills first came out, uh Zach and I were actually at an onsite together in San Francisco. And uh like we woke up one morning and they're like, "Oh, introducing skills." And we're like, "Well, this looks like every other markdown file that they provided. What what's the difference?" And um I like later that day we presented on on skills like I don't know, 4 hours later. And the the one that I built to present that was a Claude skill Claude skill uh that would analyze the like it wouldn't analyze your skill running cuz nobody had skills like 4 hours into them existing, but it would analyze, "Oh, you just did this task with Claude. Let's go through and pick out what could have been what what we what we could like encapsulate into a skill so that it can do that in an easier way." And like since then there's like meta skills and things like that that have come out where it will analyze the performance of actually how you're using Claude or how you're using the skills in Claude. And then it can use that to feed back in just like Zach was saying, just by looking at those JSONL files, they're these logs of like the conversations that you're having with Claude. And uh that can inform it on how to improve things. So for example, like in the repo roast, uh if it's kind of being wonky about how it's pulling in get information, adding in like the the bang with uh like the specific get command that you want it to run to get log information, that's a way