doesn't work. You have to change the problem definition. So we've been thinking about this and what we developed is a new way to test is called the audit task. Again, suppose you have two different bugs, but you flip the question from just find a bug to find all vulnerabilities discovered. At this point, the LLM has the freedom to find multiple bugs and submit multiple proofs of vulnerabilities. And it may be full proofs of vulnerabilities for bugs you know about and bugs you don't. You run all vulnerabilities through your oracle. And this is where it's very important to have a deterministic grader. So for here, for example, there's two vulnerabilities, it gives us two inputs that crash both vulnerabilities. And part of this grader now have to uniquify them to show that two different vulnerabilities are triggered. Now if we didn't know about vuln two, this also gives us the opportunity to increase our ground truth. We haven't told the vuln the LLM that we don't know about something that it found. It just gave us proof that it was able to find it. So we can normalize the set of known vulnerabilities at that point to be something like D star and calculate the the precision and recall for the model across multiple vulnerabilities. For example, recall is the number of known that it found over the total set and precision is the number of found over the submitted. What this prevents the model from doing and it essentially balances is the ability for it to go find unknown vulnerabilities, but also prevents you from prevents the model from just spamming. You don't want it to give you a bunch of things that aren't vulnerabilities, like for example, giving us POV in four that doesn't trigger anything. You need to disc you need to prevent that. And we found that this precision versus recall is the way to balance those two competing goals. So when you do this this way, you have an open world grading. Instead of trying to define one problem that's perfect, you can give it a real open-source task that can have multiple vulnerabilities, even though those that you don't know about. You post-hoc, since you're asking for a proof of vulnerability, you can then go say what is the total set found of those known and unknown, and you can score precision and recall and normalize both, so they're multiplicative. It won't just keep finding the same easy bug. You add as I said, it's open world, so