Has the Bitter Lesson Come for AI Detectors?

Vals built its AI detection benchmark around a problem most such tests run into: public human writing has largely been absorbed into model training, and the origin of newly written text is hard to verify. So it used private documents written before November 2022, mostly from its own staff plus a small number of crowdsourced ones, and had a suite of models rewrite them in a similar style. A rewrite counted as AI-assisted only if more than half the original text had changed. The newest OpenAI, Anthropic and DeepSeek models were left out of the rewriting so they could be judged fairly as detectors.

Claude Opus 5.5 from Anthropic led with 98.38% balanced accuracy, meaning accuracy on a set that is half human and half AI, followed by GPT-6 Astra from OpenAI at 95.75%. Both scored above every dedicated detection product evaluated, a group that included GPTZero, Pangram, Sapling, Copyleaks and Originality.ai, and Astra cost less per document than Opus.

The more useful finding is what a headline score hides. Catching rewrites was harder than recognizing human writing: the top four systems correctly classified 98.8 to 100% of human documents but only 87.9 to 96.8% of rewrites. Among the detectors, GPTZero never misjudged a human document but caught only 56.3% of rewrites, while Sapling caught more rewrites, 72.9%, and wrongly flagged nearly one human document in five, correctly classifying 82.2%. One tool misses AI text and the other accuses people, so the two error rates matter separately.

When Vals had Astra and Opus do the rewriting themselves, 73 of 100 attempts changed enough text to count, and frontier models were often able to produce edits no detector reliably caught; Pangram was fooled more easily than the models that wrote the rewrites. Vals stresses the limits: it chose an uncontaminated set over size and variety, is unsure how results generalize, and says its deliberately adversarial prompts should not be read as practical accuracy rates for detectors in everyday use.