What Does Osworld Tell Us About Ais Ability To Use Computers

OSWorld is one of the main benchmarks used to claim progress in computer use. A model gets an instruction and an internet-connected Ubuntu virtual machine, acts by writing pyautogui code that drives the mouse and keyboard, and is graded only on whether the machine ends up in the target state. In an October 2025 review for Epoch AI, Florian Brand and Greg Burnham categorized every task, one author sorting each task and the other checking a sample, to work out what a high score would mean. Their answer is that saturation would show a model can do simple, realistic tasks in Linux using popular open-source applications, such as adding page numbers to a document or exporting a CSV from a spreadsheet. Most tasks take fewer than ten steps. Only about 12% need more than 20 steps and 5% more than 50.

The bigger finding is that much of OSWorld does not test the GUI skills that 'computer use' brings to mind. Epoch counts about 15% of tasks that need only a terminal and another 30% where terminal commands and Python scripts can replace much of the intended clicking. Spreadsheet tasks are the clearest case, since packages like openpyxl can edit files directly. On the official leaderboard, agents with code execution score considerably higher than GUI-only agents, and Epoch even observed Claude Sonnet 4 installing openpyxl and pandas in a scaffold that did not provide code tools. Many instructions are also borderline ambiguous, and the authors found that working out what was meant was often the hardest part of a task. A model with every needed computer skill can still fail by making a reasonable but wrong guess.

The benchmark also moves underneath its scores. A major update in July 2025, released as OSWorld-Verified, changed most task instructions and evaluation functions, and about 10% more instructions have changed since. Epoch notes that fixing tasks that had been unnecessarily hard or impossible can make models look as if they improved when they did not. Another roughly 10% of tasks depend on live websites, so they can get harder or become impossible as those sites change. About 10% have serious errors such as wrong gold answers or graders that are too strict or too lax, a rate Epoch calls on par with many benchmarks.

None of this makes OSWorld useless. Epoch says state-of-the-art models went from less than 10% to almost 70%, and that much of that gain reflects better computer use. The caution is about comparisons. A gap between two scores, or between this year's score and last year's, could come from GUI ability, terminal and coding skill, better reading of vague instructions or edits to the benchmark, and the score alone cannot tell you which. These observations describe the version of OSWorld Epoch reviewed in autumn 2025.