Intel / article

Cross-Benchmark Generalization in Long-Horizon Agents

Source: arxiv.org
Author: Sushant Mehta, Logan Ritchie, Liudas Panavas, Edwin Chen
Date: Published Aug 4, 2026

Context

In one experiment, training an on extended tool-based tasks was followed by higher software benchmark scores even though those training tasks contained no software-engineering work. Researchers adapted Qwen3.5-122B-A10B using demonstrations followed by reward-based training on 363 tasks. Their reported comparison found gains of 5.8 percentage points on Pro and 2.8 points on Terminal-Bench 2 against the starting model.

This supports transfer beyond the training task domain in that experiment. It does not isolate which training stage caused the gain, and the authors report only one full training run. External evaluation tasks were excluded from post-training, although benchmark performance informed initial model and teacher choices.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.

Key quotes

“We argue that the discriminating question is behavioral, namely how a trained agent acts, and that cross-benchmark transfer is the right place to look for it.”

“These results provide descriptive evidence that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain.”

“a policy can get rewards by exploiting environment-specific regularities (tool schemas, grader parsing, task templates) rather than by acquiring transferable skill”

Read the source arxiv.org