Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd
Source
AI Engineer
Author
AI Engineer
Date
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
Offensive-capability evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → has no clean ground truth, and this is a concrete curriculum-plus-measurement approach.
Key quotes
“hacking is really a ladder. And this is what actually matches cybersecurity so well to reinforcement learning.”
David Brumley
“Crashing a program is different than hacking it. You can't go steal someone's IP by simply crashing a program.”
David Brumley
“don't define the task by a single bed, let the program define the task”
David Brumley
“if you could give Chrome to an LLM and it could come up with a zero-day, you would essentially be able to hack nation-states at that point”
David Brumley
“you can't judge trust the LLM that you're teaching to be a judge”
videoTraining Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging FaceAI Engineer
videoThe Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, RexmoreAI Engineer
articleOpenAI’s accidental cyberattack against Hugging Face is science fiction that happenedSimon Willison