OSWorld
os-world.github.io- Category
- Developer Tools
- Rank
- No. 868Tools index
Previous survey · No. 849 ·
- Listed in
- #20 Find AI benchmarks
- Type
- TOOL
- Builder
- xlang-ai
- GitHub
- 3.1k stars
- Latest release
- v0.1.16
- Date
About
Benchmarks multimodal computer-use agents on open-ended tasks across real desktop applications and operating-system environments.
What it does
OSWorld runs an agent inside a resettable desktop, feeds it screenshots or accessibility data, executes mouse and keyboard actions, then checks whether each requested outcome was achieved. It records trajectories, screenshots, videos, logs, and task results for later analysis.
Why it's ranked here
OSWorld is compelling when evaluation must reflect full desktop behavior instead of isolated prompts. It combines repeatable machine setup, several observation modes, explicit task evaluators, rich run artifacts, and parallel infrastructure options. That breadth comes with substantial operational weight and configuration risk.
What's good
Runs capture the evidence needed to debug failures: individual actions, model responses, screenshots, rewards, logs, recordings, and final results. Tasks can use screenshots, accessibility trees, both together, or tagged screen elements. Provider support spans local virtualization, containers, and hosted sandboxes, while result summaries break performance down by application and category.
Tradeoffs
Setup is infrastructure-heavy. Local use requires VMware or VirtualBox, while efficient container use depends on KVM. Some tasks also require Google credentials, OAuth configuration, or proxies. Interrupted container runs may leave resources behind. Packaging metadata disagrees on the minimum Python version, and the dependency surface includes many large machine learning, document, browser, cloud, and media packages.
How to use it well
Use OSWorld for controlled experiments comparing computer-use agents across identical desktop tasks. Start with the quick interaction check, choose an observation format and action space, then preserve complete result artifacts for failure analysis. Parallel providers suit larger evaluation batches. It does not replace agent training, model hosting, credential provisioning, or lightweight unit tests for individual UI components.
Technical notes+
README.md documents local VMware and VirtualBox setup, Docker with KVM, Modal, Daytona, AWS, CLI-driven experiments, credential requirements, and saved artifacts. run.py configures the single-environment evaluation loop, although its own comment marks that route almost deprecated. lib_run_single.py resets the environment, waits for readiness, executes agent actions, writes screenshots and trajectories, evaluates outcomes, logs completion, and records video. show_result.py aggregates per-domain and category results. monitor/main.py exposes a Flask monitor over task state and artifacts. mm_agents/agent.py supports screenshot, accessibility-tree, combined, and tagged-element observations. setup.py packages desktop_env for Python 3.10 or newer and installs Playwright browsers, while pyproject.toml packages osworld for Python 3.12 or newer, creating a notable metadata mismatch.
Observed
- License
- Apache License 2.0
- Primary language
- Python
- Python requirements
- setup.py requires Python 3.10 or newer; pyproject.toml requires Python 3.12 or newer
- Installation surface
- Repository installation from requirements.txt, plus a desktop-env package install without benchmark tasks
- Interfaces
- Python library environment, command-line evaluation scripts, result summarizer, and Flask monitoring interface
- Environment providers
- VMware, VirtualBox, Docker, Modal, Daytona, and AWS are documented
- Guest systems
- Ubuntu and Windows are documented for Docker experiments
- License packaging classifier
- setup.py classifies the package as Apache Software License
Read from README.md, setup.py, pyproject.toml, requirements.txt, run.py, quickstart.py, show_result.py, lib_run_single.py, lib_results_logger.py, gemini_action_parser.py, monitor/main.py, mm_agents/agent.py, desktop_env/actions.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.