CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
- Source
- arxiv.org
- Author
- Anjali Kantharuban, Jonas Mueller
- Date

Context
A simulated user can sound human and still fail to make an fail the way real users do, and an arXiv paper proposes a way to calibrate for that. Many agent benchmarks use a playing the user in multi-turn conversations. The authors argue existing simulators match human style on the surface but lack outcome , meaning agreement with real success rates and failure patterns when real people use the same agent.
Their method, Calibrated User (CUE), encodes observed sessions into continuous representations, samples new ones, and decodes them into persona instructions that steer a language model to act as the user, without training. On the benchmark τ²-Bench, they report that CUE simulators made fewer simulator-attributed errors and reproduced real-user failure modes, aggregate success rates and outcomes for specific task-user pairs more faithfully than other persona-based methods.
Fit mostly on customer support interactions, they say, the simulators also carried over to document creation, math tutoring and casual conversation, and across simulator models without retraining. The abstract gives no numeric margins. A related item covers another approach to simulator realism.
- The authors find that existing user simulators match human style but lack outcome calibration: they do not reproduce the success rates and failure patterns observed when real users talk to the same agent.
- CUE encodes observed sessions into continuous embeddings, samples from that space, and decodes samples into persona commands that steer an LLM to act as the user, with no simulator training required.
- On tau-squared-Bench, the paper reports CUE-steered simulators made fewer simulator-caused errors and better matched real-user failure modes, aggregate success rates and per task-user outcomes than other persona-based methods.
- Though fit mostly on customer support interactions, the simulators reportedly generalized to document creation, math tutoring and casual conversation, and kept working across different simulator LLMs without refitting CUE.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
articleGAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented AgentsUmesh Bodhwani, Thanh Tran, Kai Wei
articleBeyond Surface Style: Aligning Multi-Turn User Simulators with Behavioral ConsistencyGeng Chen, Ruotong Pan, Zhirui Yang, Qiqi He, Jiawei Chen, Zhang Yunfei, Chongyuan Chen, Minxuan Lv, Zheng Yang, Win-Bin Huang, Xiangyu Wu, Wenwu Ou
articleBounded Personas Match Retrieval on Classification but Not Regression for a Frozen AgentJaeHa Yoon, Minjun Park, Seoyeon Kim, Jiwoo Lee, Hyunwoo Choi, Dohyun Kang
Checking sign-in…
Loading comments…