Judge moderation on two end-to-end numbers, not classifier accuracy: Usefulness (share of turns showing a helpful, non-harmful reply) and Harmful Exposure (share showing a harmful one). Latency and errors are diagnostics, not the score.
Placement is a real tradeoff with no winner. Filtering only the model's response gave the highest usefulness; filtering both the user input and the response gave the lowest harmful exposure.
Rewriting a flagged response instead of blocking it recovered most of the blocked traffic, at the same measured harmful count as blocking. The rewrite generalizes the triggering language while keeping the benign intent.
Routing with a small probe classifier cut route-and-generate time substantially versus asking an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → to route, at comparable outcomes. The routing decision does not need a big model.
Rewrites have a quiet cost: in sensitive domains, the softened answer sometimes drops safety-relevant support information the user actually needed.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
It reframes moderation as a placement-and-recovery design problem rather than a classifier-accuracy problem.
Key quotes
“Content-moderation classifiers are usually evaluated in isolation, but deployment requires choosing where to intervene and what follows a flag.”
“At the evaluated operating points, Response only achieves the highest filter-only Usefulness in both settings, while Input + response achieves lower Harmful Exposure.”
“Replacing Response only blocking with Response + rewrite recovers most blocked traffic and yields the same observed Harmful Exposure count as Response only blocking for the selected configuration; this equality is not an equivalence result.”
“These results support comparing moderation configurations under deployment-specific safety and latency constraints rather than applying a universal placement rule.”
