Intel / article

Adversarial Attacks on LLMs

Source: lilianweng.github.io
Author: Lilian Weng
Date: Published Oct 25, 2023

Context

Using one AI model to test another creates a second problem: deciding whether a test actually exposed a failure. Lilian Weng's adversarial-attacks survey describes model-based red-teaming, in which a model generates challenging inputs, a target model answers, and a classifier judges whether the answers are harmful.

That classifier's score can then serve as the reward for training the test-generating model, so its judgments influence which attacks the process learns to produce. The survey's description assumes a good-quality classifier. This makes the quality of that evaluator central to the method: generating more test inputs is useful only alongside a credible way to recognize failures.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
jailbreak — A prompt crafted to make a model ignore its own guidelines — usually through roleplay, hypotheticals, or encoding rather than a direct request.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.

Key quotes

“Adversarial attacks are inputs that trigger the model to output something undesired.”

“UATs probably exploit biases effectively in the training data that gets baked into the global model behavior.”

“It is impossible to build a perfect classifier on detecting harmful content and any biases or flaw within this classifier can lead to biased attacks.”

“There is a clear tradeoff between sample diversity and attack success rate.”

Read the source lilianweng.github.io