AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan
- Source
- youtube.com
- Author
- Latent Space
- Date
Context
Zico Kolter and Matt Fredrikson, Carnegie Mellon researchers, describe Gray Swan to Latent Space as a company that tests AI models for security weaknesses, especially when an attacker wants an agent to leak data or misbehave. They say they use two methods: a community of about 15,000 red teamers who compete for prizes set by lab sponsors, and a family of models trained for automated .
One founder says that in the latest experiments their automated system, Shade, breaks models better than human red teamers, citing a recent human-versus- competition, with the caveat that this holds for a fixed time window and task set and is not superhuman. No data is shown. They say they helped assess a preview of Anthropic's Mythos model for indirect , where an agent reads untrusted content and is hijacked from its goal. Frontier models make poor automated attackers, they add, because safety makes them refuse, so they train their own.
- red teaming — Deliberately attacking your own AI system to find what makes it fail before someone else does.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
- prompt injection — An attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
Checking sign-in…
Loading comments…






