Intel / article

Evaluating OpenAI's Privacy Filter: Cross-Lingual, Cross-Domain PII Detection Across 42 Benchmarks

Source: arxiv.org
Author: Rohith Uppala
Date: Published Aug 5, 2026

Context

OpenAI's Privacy Filter performed unevenly across types of personal information in an independent . The revised study reports stronger detection of regularly structured details such as email addresses and phone numbers, with weaker results for person names and addresses. This matters because a filter that handles predictable formats well can still struggle with information whose form varies across languages and cultures. Revised abstract.

The evaluation covers 32 benchmarks across fourteen languages and five domains. It also reports sharp declines on general named-entity recognition tasks, which differ from tests specifically labeled for personal-information detection. The supported conclusion is that performance depends on the kind of text and labeling task; one aggregate result cannot describe all intended uses. Evaluation scope.

Key takeaways · AI-distilled

First independent eval of OpenAI Privacy Filter (1.5B) across 42 benchmarks, 22 languages, 5 domains: it beats Presidio and XLM-RoBERTa on PII-annotated benchmarks (F1 0.855 on AI4Privacy) but loses to XLM-R on all 13 Indic and non-Latin NER languages.
It collapses when PII is embedded in narrative prose - F1 down to 0.04, with Arabic at 0.04 and Cyrillic at 0.03 - so do not rely on it for free text in non-Latin scripts.
Strongest on structurally regular PII (email 0.78, phone 0.76), weakest on culturally variable types (person 0.40, address 0.49); recall-biased on medical, legal and support PII (precision 0.31-0.54 vs recall 0.70-0.85).
GPT-4o still leads on medical, legal and financial PII while OPF leads on structured synthetic PII and customer support - detector choice should be per-domain, not global.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Key quotes

“OPF degrades sharply when PII is embedded in narrative prose: F1=0.04--0.57 on NER benchmarks and collapse for non-Latin scripts (Arabic: 0.04, Cyrillic: 0.03).”

“GPT-4o leads on medical, legal, and financial PII (SPY: 0.643 avg, Gretel: 0.527), while OPF leads on structured synthetic PII (0.71 avg) and customer support (0.60).”

“Error analysis shows OPF is strongest on structurally regular PII types (email: 0.78, phone: 0.76) and weakest on culturally variable ones (person: 0.40, address: 0.49).”

Read the source arxiv.org