τ²-bench

Why it's ranked here

The framework covers unusually varied evaluation surfaces within one package: text, realtime audio, retrieval, and reinforcement learning. Its domain registry, configurable runs, saved trajectories, retries, and concurrency make it credible infrastructure, though reproducibility demands careful version control.

What's good

Domains bundle policies, tools, tasks, and optional user tools, which keeps scenarios structured and extensible. The runner separates simulation, construction, and batch execution. Batch runs support checkpointing, resumption, retries, worker processes, provider limits, seeded execution, and selectable task splits.

Tradeoffs

The installation is substantial beyond core text evaluation. Voice requires a large optional dependency set plus system audio tools, while knowledge and reinforcement learning need separate extras. Python support is limited to 3.12 and 3.13. A grading correction made older banking knowledge results noncomparable.

How to use it well

Use it when developing or comparing customer service agents that call tools, especially when repeatable domain tasks and saved trajectories matter. Start with core text runs, pin the repository state, then add voice, retrieval, or Gym support selectively. It does not provide a general production agent platform.

Observed

License

MIT

Primary language

Python

Python support

Python 3.12 and 3.13

Packaging

Hatchling build backend; installation uses uv with optional extras

Interfaces

Command-line interface, Python library, and optional Gymnasium-compatible environment

Evaluation modes

Turn-based text and full-duplex voice

Built-in domains

Mock, airline, retail, telecom, and banking knowledge

Test structure

Pytest targets are separated for core, voice and streaming, knowledge, Gym, and full-suite runs