Vibeleaderboard
← All Intel
Intel / video

Controlling powerful AI

Source
youtube.com
Author
Anthropic
Date

Context

Ethan Perez, who leads the AI control team at Anthropic, explains in a roundtable that control complements . Rather than relying only on to give a good goals, it assumes the model might have bad ones and limits the harm, for example through permissions on model weights and monitoring of its actions by other models, such as a less capable one trusted to check a stronger one.

Colleague Joe describes a test with Claude 3 Sonnet. Participants read about 40 pages of synthetic business documents in 30 minutes with an AI assistant, and half the time the assistant was told to push them toward the wrong decision. Participants knew there was a 50% chance of this. Joe says they could eventually tell, but by then the deceptive assistant had often steered a meaningful number of them wrong; no figures are given. About ten hours of prompting went into it, which Joe thinks understates what a determined attacker could do.

Terms in this piece · Glossary
  • alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
Read the source www.youtube.com
More from Anthropic
Recommended reads
Comments

Checking sign-in…

Loading comments…