Baseline report, branch per iteration, global memory file

iterations that we want to run. Um of course, the way we create the baseline data is, you know, we just run the evals once and we generate basically a document and this is super helpful because it gives the coding agent the information about the current status of the system. Then this generates a baseline report which contains, you know, all these informations. It contains what are the cases, it contains summary, what's working, what's not working. So basically, our coding agent it's going to have a clear picture of what's happening right now. And so, it will try to understand, okay, how do I improve this system from this moment? And so, from now on, we can start with the first iteration. So, every iteration starts with a new branch. We create an hypothesis. Um the system changes the agent to implement that hypothesis. We run the evals. We run our eval suite. And we generate a reports.md file, which contains everything that happened after we run the evals. We update the memory file, uh which is like a global memory file across all the runs. And if the metrics improved, then we continue from this branch. Um if the metrics didn't improve or we have a strong regression or something bad happened, uh then we roll back to the previous branch. Of course, um the generated hypotheses are based on what the agent reads when it starts the investigation, which is the memory file, uh the reports file, so it has access to pretty much everything. And that's what happens when we run multiple iterations, right? So, let's assume that in the first case, I create a branch, I create an hypothesis, uh I run the evals, everything goes well, so I get an improvement, so I am fixing one failure mode. Um therefore, I continue from that branch. I create a new one so that I can keep track of every change. Um I create the new that new branch. I try new hypothesis. This time, it didn't really improve, so maybe I get a regression, maybe things didn't improve really. Uh so, what I do is I roll back to the previous branch, so the first branch where we had a real improvement in the system. And then from here, I create a new branch, and I try a new hypothesis, and so on and so forth. At the end, what we're building is a full change log of all the changes we have done, um every improvement in improvement or regression as we can see here on the right. We are seeing all the hypothesis that we are doing and of course, if we want we can also see all the hypothesis because every hypothesis is building a report. So, let's assume that something didn't really go well in terms of evals. Maybe the evals didn't improve after an hypothesis, but maybe that hypothesis was promising, right? Maybe the agent was onto something, but it just didn't implement the system the change in a correct way, right? What we can do is as humans, we can just go back into the hypothesis, read, understand what the agent was trying to do, and then maybe steer it in the right direction next time. Of course, this is the real task on a on a real agent that we did have. And as we can see here, the baseline accuracy was 67% um but then in something around 10 iterations, we managed to reach 86% in our evals without actually cheating because it found edge cases, it improved the system prompt, it improved the tool descriptions to catch more edge cases, and it also fixed some tools logic. So, that's a real agent that is now running in production and actually this system found ways to improve it more and more.