← All IntelClip / CybersecurityBenchmark results: only GPT-class models solve at k1/k5
From Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face · ≈12:16
“The benchmark right now is incredibly hard.”
“That's why the partial graders are so critical to be able to really see what the model is able to do and what and how deep within exploitation chain they can get.”
“If every model in the world could get really really really good at doing this and very fast, that should give a lasting defense uh and capability to the defenders that the attackers simply don't have right now.”
What’s in it
- Shows an internal AI security benchmark called Bach in action
- Reveals only GPT-5.5 can complete full exploit chains
- Explains why partial grading matters for evaluating hacking depth
Clip transcript
we don't get uh there. And I I know we're basically out of time. So what we're seeing here is what's called Bach. It's our internal system because it's an orchestrator. This is how we run our actual eval. Um what we see is the actual results of the benchmark. The benchmark right now is incredibly hard. There's only one solve at K1. Um and then at K5 there is um it remains only GPT and the public models is is able to solve this. That's why the partial graders are so critical to be able to really see what the model is able to do and what and how deep within exploitation chain they can get. I'm going to load quickly the sort of the way we think about these environments which is because we're looking for performance over time. We really measure how capable is the model at making specific leaps. And so what you'll see is a results of exploitation on specific one of our environments. And then if we zoom in then you can really see how sort of GPT 5.5 is the only model that's able to make this leap. The model other models sort of have been able to reason across everything. If we look at the discovery phase they do capture nearly all the different information they need and they never are able to make the leap into what is the exploitation they need to do. This is exactly the type of capability that we believe. If every model in the world could get really really really good at doing this and very fast, that should give a lasting defense uh and capability to the defenders that the attackers simply don't have right now. Um and then
Comments
Checking sign-in…
Loading comments…