WebArena
webarena.dev- Category
- Developer Tools
- Rank
- No. 1903Tools index
Previous survey · No. 1799 ·
- Listed in
- #21 Find AI benchmarks
- Type
- TOOL
- Builder
- web-arena-x
- GitHub
- 1.6k stars
- Latest release
- v0.2.0
- Date
About
Realistic, reproducible web environment for evaluating autonomous agents on long-horizon tasks across functional websites.
What it does
WebArena gives agent developers a self-hosted browser lab with shopping, forum, code-hosting, maps, and encyclopedia sites. Agents receive HTML, accessibility-tree, or image observations, issue browser actions, accumulate trajectories, and face task-specific evaluators that mark outcomes as pass or fail.
Why it's ranked here
Its strongest case is controlled agent experimentation: site state, authentication, tasks, prompts, actions, traces, and outcome checks live in one system. The canonical repository is credible for paper reproduction, but its own documentation recommends AgentLab for parallel runs, broader benchmark integration, unified reporting, and better edge-case handling.
What's good
The Gym-like interaction model makes experiments easy to reason about. Observation modes cover structured text and screenshots. The runner renews login cookies, limits stalled agents after repeated parsing failures or actions, records rendered trajectories, optionally saves browser traces, and supports both OpenAI and Hugging Face model endpoints.
Tradeoffs
Setup is substantial. Reliable evaluation requires hosting several websites, configuring seven service addresses, generating task configurations, preparing authentication cookies, and resetting state after the full suite. Dependencies pin older Playwright and OpenAI clients. Python requirements conflict between the installation guide and package metadata. The provided runner also reads a screenshot-rendering option that its argument parser does not define.
How to use it well
Use it for research teams comparing browser agents under controlled tasks, especially when reproducing the accompanying paper or inspecting failures through trajectories and traces. Start with the walkthrough, then move to self-hosted sites for valid experiments. Prefer AgentLab when parallel execution or combined benchmark reporting matters. It does not cover terminal use or coding tasks.
Technical notes+
setup.cfg packages browser_env, agent, evaluation_harness, and llms, declares Python >=3.7, <4, and configures pytest plus strict mypy, while README.md instructs Python 3.10+. browser_env/envs.py implements ScriptBrowserEnv on Gymnasium and synchronous Playwright, with HTML, accessibility-tree, and image observations. run.py supplies the argparse evaluation CLI, early-stop checks, cookie renewal, evaluator routing, HTML rendering, and optional trace archives; it references args.render_screenshot, although config() does not declare that argument. agent/agent.py includes prompt-driven and teacher-forced agents. llms/utils.py routes OpenAI chat or completion requests and Hugging Face endpoint requests. scripts/html2json.py converts rendered evaluation output into JSON conversations with extracted images.
Observed
- Primary language
- Python
- Packaging
- Setuptools package with editable pip installation and a development extra
- Interfaces
- Python library plus command-line evaluation runner
- Browser layer
- Gymnasium environment backed by synchronous Playwright and Chromium
- Model providers
- OpenAI chat and completion APIs, plus Hugging Face text-generation endpoints
- Quality tooling
- Pytest configuration, strict mypy configuration, pre-commit, Black, and runtime type checking
Read from README.md, setup.py, setup.cfg, requirements.txt, run.py, minimal_example.py, llms/utils.py, agent/agent.py, llms/__init__.py, agent/__init__.py, llms/lm_config.py, llms/tokenizers.py, browser_env/envs.py, browser_env/utils.py, scripts/html2json.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.