>> So, now that we spoke about how we eval the routing, I want to talk about how do you get the first version of the model out. For our use case, we consider human labels as the golden source of truth. And this is what we want to align our models to. The way we do about this is we go collect a dataset which is representative. So, you know, different cuts, geographies, dish type, image quality type. Send it to our human labelers and give them a very objective guideline to label on. This is to remove any subjective biases or any noise coming in from human labelers. Once we've got that system set up is when we start tuning our model. We take our agent, we go ahead get output from the agent, compare it to your golden dataset, evaluate if it's good enough to ship, if it meets your guardrail metrics, you go ahead and ship it. If not, then you go tune and you keep doing this until you meet your guardrail metrics. For routing, our guardrail metric is recall. We don't want any bad image to slip through our system. Here are some examples of the failures we've seen. Uh on your left you see a very good image of cheeseburger. Uh on the right you notice that the routing agent actually failed this. It said the technical is low ball and it will go send this image for enhancement. Now there's two challenges when you send this image for enhancement. Firstly, you pay the compute cost for a zero quality lift from this image. And secondly, uh there is a risk of degrading this image given it's already such a high quality image. And on the other end of the spectrum, you have a recall miss. So on your left you have an image with six chicken wings and on your right if you notice the dish name, it says eight pieces chicken wings. And your routing agent approved this image. That means So now there's a risk here if you send up send this image for enhancement and you only see six chicken wings, there's a chance your model's going to hallucinate these two extra wings to to match the description. And that's also an that's a the cut we take at our faithfulness metric that Jay earlier showed us. So the meta point I'm trying to get here is you've trained your offline model, but there will be long cases where your model is going to continue to fail and the static model will not work in the real system. You need a way such that your prompts, agents, system itself is evolving over time. And that's what we've done uh for our system as well. And I'm talking more from the routing perspective, but every component in our system is able to tune itself uh for any drift online. So what we do is we sample production data at regular cadence, uh send this to the human labelers with the same guidelines that we have seen before. Once you've got that data, we compare our agents' output with the output we got from the labelers and see if there's a mismatch. If there's a mismatch, we have an umbrella diagnosis agent which takes in the feedback, localizes where this issue is happening, and and and triggers our auto-tuning pipeline. Once we tune this agent, we go and benchmark it against our golden data set that we saw earlier, and if we pass our golden data set on the metrics that we had designed, we go ahead and ship this model. Uh if not, then you kind of keep iterating. And this happens on a regular basis on production data set. Um the beauty of this is this is completely config driven and doesn't require human in the loop. Your diagnoser agent can write your config and trigger the auto-tuning pipeline here. And this is what will keep your model sharp over time. You will have one static model with the offline, but this is what is going to keep your system alive.