What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation
Source
Vishisht Choudhary, Lukas Schmidt, Anne Zo\"e Kenntner, Feras Skhab, Michel Osswald, Jens Ernstberger
Author
Vishisht Choudhary, Lukas Schmidt, Anne Zo\"e Kenntner, Feras Skhab, Michel Osswald, Jens Ernstberger
Date
Key takeaways · AI-distilled
The tell is a missing signal, not clever reasoning: Playwright never emits the raw pointer-move and wheel-delta streams a physical input device produces. That absence survives trajectory manipulation, including replayed real human cursor data.
Finding the minimal feature set took brute force — every subset of size 1 to 5 across 9,401 gradient-boosted models. The winning pair was mouse event rate and teleport click ratio, at 0.994 AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → precision.
The signal is redundantly encoded: drop teleport click ratio entirely and agent detection still holds at 100%. A single feature is degenerate, hitting perfect recall only by collapsing into always predicting "agent".
The fix for the binary blind spot is the label space, not better features. Adding an explicit agent class produced per-class agent F1 of 1.000 in all 30 runs; five features separate all three traffic classes at macro-F1 of 0.99 or better.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
If you are building or deploying browser agents, this quantifies exactly why they get flagged — automation frameworks synthesize discrete click events without the continuous pointer-move and wheel-delta streams of a real input device, and two features exploit that gap with 100% agent recall even against GAN-generated human-like trajectories. It also shows that today's binary human/bot detectors let a third of AI agents through, which matters if you run a site that cares about agentic traffic.
Key quotes
“Bot detectors deployed at scale treat traffic as binary: human or bot.”
“The discriminative signal is a browser-automation artifact, not evidence of agent reasoning: Playwright does not emit the raw pointer-move and wheel-delta streams a physical input device produces, and this absence signature survives trajectory manipulation.”
“Two features robustly isolate agents; five separate all three traffic classes at macro-F1 $\geq 0.99$.”
