First independent evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → of OpenAI Privacy Filter (1.5B) across 42 benchmarks, 22 languages, 5 domains: it beats Presidio and XLM-RoBERTa on PII-annotated benchmarks (F1 0.855 on AI4Privacy) but loses to XLM-R on all 13 Indic and non-Latin NER languages.
It collapses when PII is embedded in narrative prose - F1 down to 0.04, with Arabic at 0.04 and Cyrillic at 0.03 - so do not rely on it for free text in non-Latin scripts.
Strongest on structurally regular PII (email 0.78, phone 0.76), weakest on culturally variable types (person 0.40, address 0.49); recall-biased on medical, legal and support PII (precision 0.31-0.54 vs recall 0.70-0.85).
GPT-4o still leads on medical, legal and financial PII while OPF leads on structured synthetic PII and customer support - detector choice should be per-domain, not global.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
OpenAI's Privacy Filter beats existing tools on structured PII like emails and phone numbers but fails badly on non-Latin scripts and narrative prose, while GPT-4o still leads on medical/legal/financial PII — critical to know before deploying any of these for multilingual or unstructured data.
Key quotes
“OPF degrades sharply when PII is embedded in narrative prose: F1=0.04--0.57 on NER benchmarks and collapse for non-Latin scripts (Arabic: 0.04, Cyrillic: 0.03).”
“GPT-4o leads on medical, legal, and financial PII (SPY: 0.643 avg, Gretel: 0.527), while OPF leads on structured synthetic PII (0.71 avg) and customer support (0.60).”
“Error analysis shows OPF is strongest on structurally regular PII types (email: 0.78, phone: 0.76) and weakest on culturally variable ones (person: 0.40, address: 0.49).”
