ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model
- Source
- huggingface.co
- Date
Context
The result here is that a list of plain-language rules, learned from an 's own practice runs and pasted into its , made an older model noticeably more dependable at customer-service work without any retraining. IBM Research ran the stock ReAct agent from τ²-bench, a in which an agent talks to a simulated customer, follows a written policy and must leave the store or airline database in exactly the right state. Its ALTK-Evolve tool had OpenAI's GPT-4.1 read the agent's attempts on training tasks and boil successes and failures into rules such as authenticating the customer first and getting explicit approval before any change. Nothing else about the agent changed.
The gain shows most in Pass^4, the share of tasks solved on all four attempts, which is what the τ²-bench leaderboard reports. On 40 held-out retail tasks, adding all 292 learned rules lifted average accuracy from 78.1% to 82.5% and Pass^4 further, from 57.5% to 67.5%. On 20 airline tasks, a compact 14-rule set did best on Pass^4, rising from 35% to 45%, so the right amount of guidance varied by domain. The extra prompt raised cost per retail conversation from about $0.084 to $0.139, while matched conversations got slightly shorter. Rerunning airline with GPT-5.5 as both agent and rule extractor lifted Pass^4 from 55% to 70%. IBM's earlier post explains why it targets this all-attempts measure: an agent's average success can hide how often it fails a task it solved before.
The headline leaderboard claim needs one qualification. τ²-bench scores submissions on the full task set, training tasks included, and the rules were mined from those same training tasks. The 67.5% retail figure that puts the GPT-4.1 agent ahead of every listed entry, including a model fine-tuned on synthetic retail data at 61.4%, is that full-set number, which happens to equal the held-out retail result. The held-out results above are the cleaner evidence for generalization, and they come from small task sets. These are IBM's own runs, compared against an October 12 leaderboard snapshot and submitted to the benchmark as pull requests.
- ALTK-Evolve has the same GPT-4.1 model that runs the agent distill its training-task successes and failures into short rules, appended as a markdown list after the domain policy and applied unchanged to held-out tasks, with no human annotation.
- Each guideline carries a support count (distinct training tasks it came from). Keeping only support 3 or higher left 31 retail and 14 airline rules, versus 292 and 101 when every consolidated rule was kept.
- The right amount of guidance varied by domain: in retail the full rule set gave the big Pass^4 jump (57.5% to 67.5% held-out), while in airline the 14-rule core was more reliable (45% Pass^4) than the full set (40%).
- Guidelines raised cost through prompt size, not conversation length: retail went from about $0.084 to $0.139 per conversation with all rules, yet matched solved runs used slightly fewer turns; the authors say can recover much of the cost.
- Rerun with GPT-5.5 on airline, guidelines lifted held-out Pass^4 from 55% to 70%, a larger gain than with GPT-4.1, though no mined rule reached support 3 there, so the high-support core used support 2.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
Checking sign-in…
Loading comments…