Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs

A computer-use benchmark can reward remembering an action sequence without testing whether an agent understands the screen. The research accompanying this talk reports that a script replaying recorded actions without observing the display outperformed frontier models on prominent static benchmarks.

The problem is that a predictable environment can make the same sequence succeed repeatedly. The proposed alternative, described in the talk, varies conditions such as starting screens, data and themes while checking that the resulting tasks remain valid. This makes benchmark success more informative about adaptation to changing circumstances, the ability users need when real interfaces depart from a rehearsed path.

“A script under one megabyte that never looks at the screen matches or beats the frontier model it was copied from.”

AI Engineer

“The paper goes further and proves that pass@k on a deterministic environment is exactly the success rate of that replay script, so a metric the field leans on turns out to be a formal measure of the exploit.”

AI Engineer

“Environments get the PRISM principles: privileged verification, realism, integrity checked configurations, sandboxed execution, and multifactorial variation across data, theme, and starting screen.”

AI Engineer

“DIGIWORLD instantiates them in 15 sandboxed mobile apps and 3.2 million verified configurations, generated by a compiler that produces every combination and rejects the broken ones, because a coding agent emitting a lot of software is not the same thing as a good environment.”

AI Engineer

“Naive rollouts on a single base case yield confidence intervals that actually contain the true performance around 20% of the time rather than 95%, and he prices the consequence: a 4% gap between two models, hidden under intervals that look tight, costs hundreds of thousands of dollars a month across a million tasks.”

AI Engineer