Adversarial attacks on images work because pixels are continuous and you can follow a gradient toward the failure. Text is discrete, so there is no direct gradient signal to push on, which is what makes text attacks harder to construct.
Jailbreaking is a special case of controllable text generation. The attacker is steering the model toward a target region of output space, the same machinery as style control, just with an unsafe target.
alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition → work like RLHFReinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.Full definition → installs safe behavior as a default, not a guarantee. Adversarial prompts route around the default, so safety training is a layer you test against attacks rather than a box you check once.
Terms in this piece · Glossary
jailbreak — A prompt crafted to make a model ignore its own guidelines — usually through roleplay, hypotheticals, or encoding rather than a direct request.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
RLHF — Reinforcement learning from human feedback — training a model to prefer answers humans rate as better, which turns a raw text predictor into a usable assistant.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
A rigorous, research-groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → map of how jailbreaks and adversarial prompts actually work against aligned LLMs, giving engineers the vocabulary and threat models needed to red-team and harden their own AI products.
Key quotes
“However, adversarial attacks or jailbreak prompts could potentially trigger the model to output something undesired.”
“A large body of ground work on adversarial attacks is on images, and differently it operates in the continuous, high-dimensional space.”
“Attacks for discrete data like text have been considered to be a lot more challenging, due to lack of direct gradient signals.”
