Intel / repo

tau2-bench v2.0.0 — τ-bench 2.0.0 — Multilingual and Voice

Source: github.com
Author: soham-sierra
Date: Published Oct 20, 2026

Context

τ-bench (tau2-bench) from Sierra tests customer-service agents. An with tools handles conversations with an -simulated caller in domains such as airline, retail and telecom, and is scored on whether it completes the task correctly. Version 2.0 adds language packs for six languages and a set of frozen, pinned text and voice benchmarks. Its most consequential message for anyone reading leaderboards is that its scores cannot be compared with 1.x scores (release notes). Sierra says to re-run models rather than set 2.0 numbers beside older ones, and it keeps the two on separate leaderboard tabs.

The CHANGELOG's upgrade section shows why. Checks on what the agent tells the caller now go to an LLM judge by default, with substring matching only on request. The judge for natural-language assertions moved from GPT-4.1 to a newer GPT-5.4 mini model. Lookups of names, IDs and phone numbers now ignore case, accents and formatting, and an ambiguous match now raises an error. Fifty English retail tasks and fifty telecom tasks were replaced while keeping their old IDs. The simulated caller now runs on GPT-5.5 instead of GPT-4.1, with new persona and language guidelines. Sierra says no combination of flags reproduces the 1.x setup exactly.

Scores comparable to the old leaderboard come from the `v1.0.1` tag. Even within 1.x, that release changed grading in the banking domain enough that banking scores from before and after it do not line up, so the version behind any older number matters too. A gap between a model's 1.x and 2.0 numbers mixes changes in the with any change in the model.

Key takeaways · AI-distilled

Communication checks, NL-assertion judging, tool-argument lookups and English prompts all changed in 2.0. Teams wanting numbers comparable with the 1.x leaderboard should use the v1.0.1 tag.
The default user simulator moves from gpt-4.1 to openai/gpt-5.5 at low reasoning effort, and communication checks are now LLM-judged, with gpt-5.4-mini-2026-03-17 as the NL-assertion judge.
Frozen manifests replace ad hoc task selection: text-v1 and voice-v2 each run 900 calls, voice-v2-banking 582 and voice-v2-entity 1,200, while the new intake domain tests exact entity capture across 200 tasks.
The new tau2 judges suite scores saved runs after the fact but is diagnostic only and never counts toward reward; tau2 submit prepare builds judged 2.0 leaderboard packages, and 1.x submissions are still accepted.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

Read the source github.com