General LLMs (Claude Opus 5.5 at 98.38% balanced accuracy) outscored dedicated AI-detection products on a private dataset. Similar overall scores hid very different human-versus-rewrite error rates, so check both before trusting a detector.
Key takeaways · AI-distilled
On Vals' private benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → of human texts written before November 2022 and their AI rewrites, Claude Opus 5.5 led with 98.38% balanced accuracy and GPT-6 Astra followed at 95.75%, both above the dedicated AI-detection products tested.
Catching AI rewrites was harder than recognizing human writing: the top four systems correctly classified 98.8-100% of human documents but 87.9-96.8% of rewrites, and GPTZero caught only 56.3% of rewrites despite 100% on human text.
When Astra and Opus 5 themselves did the rewriting, Vals found the edits were often not reliably detectable by any detector, and Pangram was fooled more easily than the generating models.
Vals cautions the rewrite prompts were deliberately adversarial and the dataset small and narrow, so the results set a floor rather than measure practical accuracy.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
