Harvey LAB (Legal Agent Benchmark)
github.com/harveyai/harvey-labs- Category
- AI Agents
- Rank
- No. 917Tools index
Previous survey · No. 912 ·
- Listed in
- #69 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- CLI
- Builder
- harveyai
- GitHub
- 1.4k stars
- Latest release
- v1.1.0
- Date
About
An open-source benchmark from Harvey AI for evaluating LLM agents on realistic legal work, combining a dataset of 1,671 tasks across 24+ legal practice areas with instructions, documents, and rubrics, plus an execution harness for running agents and scoring their outputs. It includes a full tutorial walkthrough covering setup, task inspection, agent runs, scoring, and report comparison.
What it does
Harvey LAB turns legal assignments into repeatable agent experiments. Each run gives a model a contained workspace, source documents, instructions, and document tools. A separate model then checks the resulting work product criterion by criterion, while static reports expose misses, document coverage, token use, latency, and cost.
Why it's ranked here
This is a serious evaluation system, not merely a task collection. It isolates agent execution, preserves full traces and metrics, supports several model providers, and makes failures inspectable. Its strict all-pass method suits material legal omissions, although results still depend on semantic judgments from another model.
What's good
The strongest design choice is criterion-level evidence. Judges receive only the deliverables relevant to each requirement, reducing unrelated context. Runs preserve transcripts, outputs, coverage, tool counts, and configuration. Model sweeps automate execution, evaluation, and reporting. The sandbox blocks network access, drops capabilities, and keeps source documents read-only.
Tradeoffs
The documents are synthetic, batch-generated, and explicitly described as imperfect. Evaluation has no golden answer and relies on model judgment, so rubric quality and judge behavior shape results. The all-pass verdict intentionally denies task-level partial credit. Setup also requires Podman, recent Python, Pandoc, provider credentials, and paid model calls.
How to use it well
Use it when researchers or legal AI teams need repeatable comparisons across models, prompts, reasoning settings, or agent skills. Start with one task, inspect its requirements, preserve the trajectory, grade it, then expand through dry-run sweeps. It does not provide a web service, database, authoritative legal answer set, or production legal validation.
Technical notes+
pyproject.toml defines the harvey-labs Python package for Python >=3.12,<3.14 and includes document, provider, visualization, and pytest dependencies. harness/run.py exposes the CLI, loads tasks, selects adapters, starts the sandbox, loads skills, and records configuration. harness/tools.py defines the six-tool closed workspace. sandbox/sandbox.py implements persistent Podman containers with /workspace/documents read-only, /workspace/output writable, no network, dropped capabilities, resource limits, and host-user ownership. evaluation/judge.py supports Anthropic, Google, OpenAI, and Mistral judges with structured JSON verdicts and retries. docs/architecture.md documents the filesystem-first run, evaluate, report pipeline, while evaluation/charts.py builds static comparison visualizations.
Observed
- License
- MIT
- Primary language
- Python
- Python support
- >=3.12,<3.14
- Install surface
- Repository setup script plus a pyproject.toml package; Pandoc is also required.
- Interface
- Command-line modules for task inspection, agent runs, evaluation, reports, comparisons, playback, and sweeps.
- Model providers
- Agent adapters cover Anthropic, OpenAI-compatible services, Google, Mistral, Baseten, and Fireworks.
- Platform support
- Podman-based operation is documented for Linux, macOS, and Windows; Windows setup requires Windows 11, WSL2, and CPU virtualization.
- Storage architecture
- Filesystem-first. No database or web service; runs and static HTML reports are stored locally.
Read from README.md, pyproject.toml, docs/tutorial.md, docs/architecture.md, docs/eval-strategies.md, harness/run.py, harness/tools.py, sandbox/sandbox.py, evaluation/judge.py, evaluation/charts.py.
What it can do
Run LLM agents on legal tasks via execution harness
Agent and task instructions/documents → Agent-generated task outputs
Score agent outputs against task rubrics
Agent outputs and rubrics → Scores
Compare scoring reports across runs
Multiple scoring reports → Report comparison
Inspect individual legal tasks
Task ID → Task details (instructions, documents, rubric)
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.