← All IntelClip / CybersecurityThe one-vulnerability assumption invites reward hacking
From Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd · ≈10:03
Explains why benchmarks built on real open-source software silently plateau: with multiple bugs present and crash-only grading, the model re-finds the cheapest one forever.
What’s in it
- Explains why benchmarks built on real open-source software silently plateau: with multiple bugs present and crash-only grading, the model re-finds the cheapest one forever.
Clip transcript
So, that's the basic setup. But, there's a problem with this. This is the way if you go look at the existing benchmarks like uh Cybex or uh CyberGym, they set up the task. But, there's actually a problem here. And that's because there's an assumption that the program only has one vulnerability. I don't know about you, but it's very rare to find a program for which you know there's only one vulnerability. So, what happens if you have two vulnerabilities here? This actually breaks a lot of assumptions in current evaluation environments. You ask the same question, "Dear LLM, can you find and exploit the vulnerability?" But, now the LLM has a lot of freedom to reward hack. For example, which vulnerability should it find? If you came in only knowing about the first vulnerability, but there's a second one you didn't know about, what do you do if the LLM thinks it found a second one? Or suppose you know two. What we found is on existing benchmarks with multiple pro- with a real OSS benchmarks, there are multiple vulnerabilities, the LLM will just continue to find the easiest vulnerability, and that really limits its trajectory as far as what it can learn. And then you have a question, if it does find a vulnerability you did not know about, well, how do you score it, right? You
Comments
Sign in to comment.
Loading comments…