Agents' Last Exam
agents-last-exam.org- Category
- Developer Tools
- Rank
- No. 1038Tools index
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web
- GitHub
- 1.1k stars
- Latest release
- runtime-qe-bgw-6.7.0-4.0
- Date
About
Evaluates agents on extended professional workflows across multiple domains, using a shared framework for running agent harnesses.
What it does
Agents' Last Exam is a benchmark plus the open runner that executes it. You pick an agent such as Claude Code or Codex, a sandbox (a Windows or Ubuntu virtual machine on Google Cloud, AWS, Alibaba Cloud, a local KVM host, or a Docker container), and a task. The runner boots the machine, stages the inputs, hands the agent only a task description and lets it work until it stops, then copies in a hidden answer key and grades the files left behind on a scale from zero to one. Screenshot, click and type are exposed to terminal agents as ordinary tools, so command-line agents can drive a desktop too.
Why it's ranked here
Grading is outcome-based, and the answer key reaches the machine only after the agent finishes, so it cannot leak into the run. The scope is wide: around 150 public tasks covering 55 subdomains in 13 industry clusters, mapped to the U.S. federal occupational taxonomy. Tasks are real work, such as filling a Form 1040 from W-2 and 1099 source forms across five variants. It comes from UC Berkeley RDI with a published arXiv paper, the framework is Apache 2.0, and the task data is CC BY 4.0, so results are reproducible by anyone with the cloud budget.
What's good
Every run is recorded in one schema: each step, tool call and observation, plus raw logs, the grade and the files produced, so a score can be audited end to end. The runner prints the full agent-by-task matrix in a dry run before spending money, filters by agent or task, and resumes on its own: completed and timed-out units are skipped, failures are retried up to a set attempt limit. The local KVM provider caches base disk images under a file lock, resumes interrupted multipart downloads, and gives each run a throwaway overlay so parallel runs never modify the base image.
Tradeoffs
This is heavy infrastructure. The recommended path is Google Cloud: create a project, attach billing, mint a service account key, copy the published images and open a firewall port to each guest. Windows tasks need paid billing, not the free trial. The local KVM provider has no GPU support, cannot load task data from a local directory, and does not cap total CPU or memory, so concurrency is your problem. Docker covers only a smaller Ubuntu subset, and the Docker executor is described in its own code as a shell-only stub. Some tasks need licensed software, and judge-based tasks need separate evaluator API keys.
How to use it well
It fits teams that build or compare agent harnesses and want a score on long professional workflows rather than coding puzzles. Start with the hello demo: the confirmation code exists only on screen in a web form, so it proves screenshots, clicks and grading all work in one run. Then move to the unlicensed task list, set concurrency against your cloud quota and model rate limits, and leave auto-resume on. To test your own harness, write a small deployer. It is the wrong tool for quick model-only evals: it measures whole agent systems on real machines, and every run costs a virtual machine.
Technical notes+
Entry point is ale_run/cli.py (invoked via ale_run/__main__.py) with run and list subcommands; run takes an experiment YAML plus --dry-run, --agent, --task and --disable-resume. Resume scans the per-unit output tree (agent/model/task/variant/timestamp) and treats statuses completed and timeout as done. ale_run/orchestration/__init__.py exposes Runner, ExperimentSpec and RunUnit. ale_run/executors/__init__.py registers SandboxExecutor (HTTP RPC to an in-guest cua-server), LocalExecutor (host-side, used by AleClaw) and DockerExecutor (shell-only stub). ale_run/environments/__init__.py wraps providers in the OpenEnv Environment shape with step deliberately absent. Tasks use cua_bench decorators for load, setup and evaluate; tasks/business_finance/taxform_4_1/main.py defines five Windows variants scored against reference JSON with an optional alternate reference. pyproject.toml pins openenv-core and cua-bench exactly and declares a uv workspace with one package per agent. docs/local-qemu.md documents the Dockur-based runner image and per-run qcow2 overlays.
Observed
- Software license
- Apache-2.0 (pyproject.toml)
- Benchmark data license
- CC-BY-4.0, per README.md
- Language
- Python, packaged with hatchling as a uv workspace
- Interface
- Command-line runner with run and list subcommands, configured by experiment YAML
- Sandbox providers
- Google Cloud, AWS, Alibaba Cloud, QEMU/KVM, Docker (Ubuntu subset), existing CUA sandbox
- Guest operating systems
- Ubuntu and Windows
- Grading
- Hidden reference staged after the agent finishes, scored between 0 and 1
- GPU tasks on local KVM provider
- Not supported (docs/local-qemu.md)
- Planned provider
- Local VMware, listed as planned in README.md
Read from README.md, pyproject.toml, ale_run/cli.py, ale_run/__main__.py, ale_run/orchestration/__init__.py, ale_run/executors/__init__.py, ale_run/environments/__init__.py, ale_run/tasks/__init__.py, ale_run/agents/claude_code/__init__.py, docs/quickstart.md, docs/local-qemu.md, tasks/demo/hello/main.py, tasks/business_finance/taxform_4_1/main.py.
What it can do
Benchmark AI agents on long-horizon professional tasks with verifiable outcomes
AI agent → Task performance results
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.