with AI agents and not just like machine learning models? The answer is yes. So, I built something which is called AutoAgent, which is basically the same idea, but applied to um actually like AI agents, right? So, basically this is a loop which is able to run the evals and then um update the code of the system, try new system prompts, create new tools, doing everything autonomously, and then check again whether things improved or not. And it actually works quite well. So, here we have a few iterations, and this is the AI agent This is the naive AI agent stat agent that we have seen earlier. And in that agent, as you can see here on the bottom left, um we have the baseline accuracy, which was 18%, and we managed to reach up to 83% um in like something around 10 uh 10 iterations. Now, of course, that's a um that that's a very naive case, right? Because we started with something which didn't have any tools and nothing else. So, it's relatively easy for the system to improve this agent. But, uh we also improved um some evals by 10% on a production agent that was already humanly optimized. So, actually the the machine uh the the agents the the coding agent found new ways that humans didn't find um to improve the agent, and we got plus 10% on some of our internal benchmarks. Now, how does it works? Actually, I I gave already like a few insights of how this works, but let's see it in details. So, basically, the core idea is that we have a coding agent, in this case Claude Code, but it can work with multiple coding agents. It builds the agent, so it writes the code of our target agent, and then the target agent is giving feedback to our Claude Code or whatever coding tool we're using as evals, right? So, it's giving some information as okay, here I got some regressions or these evals are not passing. Also, Claude Code can read the traces, the thinking traces, or like the full trace of the target agent to see whether something broke or something is not working as expected, so it can do like self-improvement. Of course, there is the human in the loop, and the human in the loop is usually in structuring the initial agent and giving as much context as we can to our coding agent about what it can do with other relevant files, what it cannot do, like just to give an example, updating the golden data sets or the scorers just to let the evals pass is not a good idea, so we want to enforce we want to tell the we want to tell the AI agent to not do that, right? So, as humans, we can we can steer a little bit our our coding agent so that it can optimize the target agent. Now, there are a few steps, right? The first step is to create an optimization job. Of course, it's very easy, everything is a markdown, we all love markdown now, so we have we can define the objective, we can define target repository, the metrics, and literally everything, right? So, in this file, we can give as much context as we want to our coding agent so that it can actually run and improve the target agent. And then the second step is to run the loop. So basically, we run the evals once, so we we we generate our baseline data. So we understand what's working, what's not working, you know, the system will generate a first report. And then we run the loop, the optimization loop. So we can see that in the first case, we get a regression, so basically the system works by creating an hypothesis. So it's tackling one class of problems at a time. It's updating the the agent and it's running the evals again. And then it's saying whether we got a regression, an improvement, or something else happened and so the agent can decide whether to roll back the change or move on. So in this case, in the first iteration, we got a regression, so we want to roll back. Um then we see a 5% improvement, which is great. So in this case, we are getting we're creating a new hypothesis, we get a 0% improvement, unfortunately, but of course the agent is keeping track of all the hypothesis and so in this case can go and create a new one and we can see a 12% hypothesis and and so on and so forth, right? Of course, we define the number of iterations that we want to run.