Vibeleaderboard
← All Intel
Intel / article

LegacyWorld: Atomicity-Aware Evaluation of GUI Agents for Legacy Workflows

Source
arxiv.org
Author
Thilo Reintjes, Sivajeet Chand, Derui Zhu, Sushant Kumar Pandey, Alexander Pretschner
Date

Context

A computer-use can fail a task and still leave damaging changes behind. LegacyWorld evaluates that distinction in older enterprise software operated through graphical interfaces. Its criterion, atomicity, requires either correct completion or failure without unintended persistent changes.

The researchers constructed 28 Windows workflows with defined starting states, goals and task-specific checks, then evaluated six hosted agents. They report different patterns of successful completion, safe failure and failures that leave changes behind. This makes the state of the records after an unsuccessful run part of the acceptance test, rather than treating every failure as equivalent.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Recommended reads
Comments

Checking sign-in…

Loading comments…