Intel / article

Reducing Toxicity in Language Models

Source: lilianweng.github.io
Author: Lilian Weng
Date: Published Mar 21, 2021

Context

A toxicity filter can suppress harmless speech as well as miss abuse. In Lilian Weng's 2021 survey, studies of detection systems found that mentions of minority identities and African American English could receive inflated toxicity scores. Reliance on particular words can substitute for understanding their use in .

That matters when a detector's judgment controls whether a conversation continues or receives a canned response. needs to examine which harmless messages are rejected alongside which harmful messages get through. Clear annotation guidance and diverse, qualified annotators help define the training signal; testing false positives checks how that definition affects people using the system.

Terms in this piece · Glossary

context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Key quotes

“Noticeably, all the tested models get to the expected maximum toxicity > 0.5 after 100 generations. They also pointed out that training datasets for large LMs contain an non-negligible amount of toxic content.”
Lilian Weng

“Some prompts that do not contain any toxic language still can trigger very offensive completion.”
Lilian Weng

“recent work has found that systems are overestimating the prevalence of toxicity in text that contains a minority identity mention (e.g., “I’m a gay man”) or text by racial minorities (e.g., text in African American English). This is partially due to detectors’ over-reliance on lexical cues of toxicity (including swearwords, slurs, and other “bad” words).”
Gehman et al. 2020

“Bad word filtering is a pretty intuitive and effective way to avoid explicit profane words in the language model generation. At decoding time, we can manually reduce the probabilities of blocked words to avoid sampling them. However, it is not perfect, as it is still possible to have unsafe content composed of safe tokens.”
Lilian Weng

“Second, there is no clearly and widely agreed-upon categorization and definition of unsafe behavior in pretrained language models. Individual perceptions could vary a lot due to different social backgrounds.”
Lilian Weng

Read the source lilianweng.github.io