Clip transcript
Now, after we've done the initial uh orchestration research and model serving research, we wanted to actually commit in code. This is just a example of what happened when we were experimenting. I was doing some initial research with Temporal and committed some activities and workflow code to make sure that we can actually replicate what we have in the legacy code base. So then, I did what we wanted to do over a number of iterations and at the time gave it to O3 to actually try to implement this code. And it did it much faster than I would be. This refactor took uh 3 hours of back and forth chatting with Incursr, but it made 10 major mistakes. So, at the time when we were going through this refactor, agentic coding was getting better and better, but it still hadn't reached the point of where it is now. And it was still a very manual process where you had to intervene and actually guide the model and manually edit or delete code. Now, I reran these benchmarks on some some modern models. So, we have Sonnet 4.6 and Opus 4.8. And things were much faster. Sonnet 4.6 with one additional iteration was able to solve the task. And with Opus, it was basically able to one-shot this problem. So, models are getting significantly better along with harnesses. And the interesting part here as well is that the way that the models interacted has changed substantially as well. Before with O3, there weren't substantial tool calls on certain categories. And then as we moved into Sonnet 4.6 and Opus, we see now that in modern harnesses, we get sub-agents, we get some of those plan calls, we get different shell commands, and we get different verifications. And overall, this process, even though the model execution was a little bit more expensive, it was a lot less manual. So, we could actually accomplish a lot more. So, if I was rebuilding the same task that I had for for this refactor, it would take around 1/5 of the time to accomplish, which is pretty good progress.