Clip transcript
used to do a lot of the changes that I'm talking about here. To be clear, when you look at the generated agents MD and Claude MD, don't assume I used those because I didn't. I was using them largely as a reference point to get things tuned how I wanted. I opened with a relatively wordy prompt on what my goals were here that I want to rethink the agents MD and Cloud MD for T3 code. I want them to better match the tone of my global agents in CloudMD. I also want to better represent the project and how agents should work. Random additional things I want to touch on. These are things I knew had been happening a lot that were annoying me. And also I wanted to pull in my glossery from my lake bed agents MD because I found that that has been super super helpful for how I communicate with the agent. I also said to match the tone from that file write up two separate drafts for an alternative cloudmd based on what I shared here and it wrote them and I didn't like them much. There were parts there that were good but it wasn't great. But I asked to do something that ended up being significantly more helpful. Can you look through my history with models like Fable, Opus, and GPT56 Soul in Cloud Code and Codeex on this machine to see what the most common mistakes are? Want to make sure we optimize to steer away from those. I want you to break down the most common failure modes and how often each model hits them. This was fun because not only did it catch specific failures that I cared about, it also categorized them by how often they happened with each model and harness. I noticed Opus 5 was aggressively killing the wrong process. Often killing the T3 code instance that it was running in and it now is confirmed with real numbers. Even though I only used it for 2 days, it had done significantly more attempts to kill processes than Fable had in all the time I use it. Here you can see all the times user data were touched. Thankfully, it was almost all readonly inspection, but we should still audit it further. PR hygiene. We noticed soul see again 40% of the time it filed PRs, it was filing drafts. Other models were much less likely to file a draft. A lot of repowide check failures where it was running things it didn't need to, taking way more resource and build time on my machine. Good call out that it had here. And it also tracked the bash error rates for the different agents that were working here as well. I will say that I did all of this with Opus because I was out of Fable at the time and it didn't do quite as good of a job auditing as both Fable and 56 did. I did have all of them do audits looking for various failures and try to collect as many of them as possible so that I could use that as a reference point. As I mentioned, I had a lot of agents do this audit with me and find all the weird failures that happened across all of my real runs on my real machines. I forgot which model made this one, but I really, really liked it. This was measuring corrections per 100 user messages per model. So, I would often tell Fable it implemented things wrong, but that's also because I use Fable to do the hardest stuff, and I often give Fable less context going in. I also apparently tell Opus 5 that its results look wrong way more often. It looks like Soul's really good here, but remember, I don't use GPT models for UI stuff. So, if I say it looks wrong, I'm probably just going to move off of Soul. For process, I actually had more problems with Fable than other models for like leaving branches stale, not fixing CI, stuff like that. Opus 48 would misread what I was asking for very often. Tool misuse. Opus was the worst by far. Opus 5 was misusing [ __ ] all the time and breaking my environment. Overbuild opus 5 and opus 48 were the most aggressive with. Apparently fable slightly worse than soul overall, but that's a universal problem. Stopping early was also a universal problem. No verify was pretty universal. Regression was rare but expensive when it happened. And only soul was doing unasked edits that I had to tell it like, "Hey, don't edit that. undo that. And then it proposed a bunch of changes to my global files that it thought made sense. And I