Shows that translation-only multilingual safety tests miss effects of local context. Teams shipping models across languages should test language and context separately.
Key takeaways · AI-distilled
ROK-FORTRESS, from Scale AI and the Korea AI Safety Institute, varies language (English or Korean) separately from geopolitical groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → (U.S. or Korean entities) across 1,235 adversarial tasks, each paired with a benign prompt to measure over-refusal.
Across nearly all 14 models, Korean-language, Korea-grounded prompts drew the least harm and English prompts the most. The language effect (about 10 points) was roughly 2.5 times the context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → effect (about 4 points).
With role-play and other adversarial wrappers removed, the Korean advantage mostly vanished: proprietary models stayed modestly safer in Korean, but five open-source frontier models became more likely to comply in Korean.
Lower Korean harm scores came partly from conservatism: models refused harmless Korean requests far more often, in some cases about twice as often, though 12 of 14 still gave less harmful substantive answers in Korean.
In four of 14 models, adding Korean context significantly weakened the Korean-language reduction in harm, which the authors say makes translation-only safety evaluations potentially misleading for real deployments.
Terms in this piece · Glossary
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
