Vibeleaderboard
← All Intel
Intel / video

AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan

Source
youtube.com
Author
Latent Space
Date

Context

Zico Kolter and Matt Fredrikson, Carnegie Mellon researchers, describe Gray Swan to Latent Space as a company that tests AI models for security weaknesses, especially when an attacker wants an agent to leak data or misbehave. They say they use two methods: a community of about 15,000 red teamers who compete for prizes set by lab sponsors, and a family of models trained for automated .

One founder says that in the latest experiments their automated system, Shade, breaks models better than human red teamers, citing a recent human-versus- competition, with the caveat that this holds for a fixed time window and task set and is not superhuman. No data is shown. They say they helped assess a preview of Anthropic's Mythos model for indirect , where an agent reads untrusted content and is hijacked from its goal. Frontier models make poor automated attackers, they add, because safety makes them refuse, so they train their own.

Terms in this piece · Glossary
  • red teaming — Deliberately attacking your own AI system to find what makes it fail before someone else does.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • prompt injection — An attack that hides instructions in content an AI will read — a webpage, email, or document — tricking it into following the attacker instead of the user.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
Read the source www.youtube.com
More from Latent Space
Recommended reads
Comments

Checking sign-in…

Loading comments…