Anthropic's August 2024 safety bounty targeted attacks that repeatedly bypass safeguards across different harmful questions. Such a reusable bypass can expose a broader weakness than a single problematic answer. The company announced rewards of up to $15,000 for qualifying discoveries in high-risk areas, including cybersecurity and chemical or biological threats.
Selected researchers would test safeguards before public deployment in a controlled environment. The program was designed to find weaknesses early enough to improve those protections; the announcement describes that testing plan rather than results from a completed evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
jailbreak — A prompt crafted to make a model ignore its own guidelines — usually through roleplay, hypotheticals, or encoding rather than a direct request.
