Vibeleaderboard
← All Intel
Intel / article

Harnesses Aren't Obsolete: They Make LLMs Compositional Generalizers

Source
alphaxiv.org
Author
MIT CSAIL
Date

Context

Stronger models do not make the harness irrelevant, this paper argues, because the harness decides what each model call actually sees, and that decides whether training on easier tasks carries over to harder ones. A harness is the wrapped around a : it gives the model its instructions and , runs the tools the model asks for, and passes the results back. The common design, which the paper calls trajectory-as-a-prompt and traces to ReAct, appends every step and tool result to one growing prompt. Train a model that way on short tasks, then hand it a much longer one, and the prompt it faces can look unlike anything it trained on. Alex L. Zhang, Souradip Chakraborty and Omar Khattab of MIT CSAIL test a different design in their paper and argue that it can keep every individual call familiar even when the task as a whole is new, a property they call locally in-distribution.

The design is the Recursive Language Model, or RLM, a harness Zhang and colleagues introduced earlier. It stores the long input as a variable in a Python REPL, an interactive code session where the model runs code and reads the printed output, instead of pasting the input into the prompt. The root model, the call in charge of the task, writes code to inspect that variable and can hand selected pieces to sub-calls: fresh model calls that see only the piece and instruction they are given. In one example from the paper's appendix, the input holds 787 trivia questions and the task asks whether questions whose answer is a place are more or less common than questions whose answer is a person. The root model, trained with reinforcement learning (it attempts tasks many times and is pushed toward the attempts that scored well), writes a short program: extract the questions, send them to sub-calls in batches of 20 to be labeled with one of six answer types, then count the labels and compare. That run used 41 sub-calls. On a test version with 6,374 questions, eight times as long, it wrote the same program and made 321 sub-calls. Each sub-call still saw 20 questions, so its job did not get harder: on this task, sub-calls from the trained model labeled 85.6 percent of questions correctly on the short training tasks and 83.7 percent on the long test tasks.

Across six tasks whose length can be varied, the authors trained Qwen3-30B-A3B, a Qwen model, only on short versions and tested it on versions 2 to 64 times longer. Inside the RLM, its score on the long tests rose roughly in step with its training score. The same model trained without a harness, reading everything in one prompt, with its context window stretched by a standard method where the long tests needed it, often scored higher in training but fell behind on the longer tests. Two other harnesses complete the comparison. One gives the model a code session but no sub-calls; it kept up on tasks a program can solve outright, such as listing a node's parents in a graph, but struggled on tasks like the trivia count that need judgment applied to thousands of items. Pi, a coding- with sub-agents, also struggled on those dense tasks. The RLM transferred best again when the subject matter changed but the solution shape did not, for example training on essay-writing questions and testing on math questions that the paper says share a ranking-style breakdown.

That transfer has a limit, and the paper's answer to it produced the headline model. When the authors trained Qwen3.8-27B with reinforcement learning on the mix of short tasks and tested it in the RLM harness on eight harder long-context benchmarks that do not share those tasks' structure, including CTC-Bench and ExpertLongBench, the average barely moved, from 46.1 to 48.5. So they trained the root model's behavior on stand-in tasks instead, a method they call surrogate harness training. Claude Opus 5.5, working in Claude Code, was given a separate public portion of each benchmark and some example runs, proposed a program that should solve it, and built synthetic tasks with unrelated content that the same program also solves. For three benchmarks, the sub-calls in these synthetic tasks returned pre-written answers instead of calling a model, which makes practice runs cheap. The authors then had the RLM solve the synthetic tasks with that program given as a hint, kept 8,015 root-model runs that a judge model and the synthetic task's scoring accepted, plus 1,465 sub-call runs for one benchmark, and fine-tuned Qwen3.8-27B to imitate them. That step is supervised fine-tuning, not reinforcement learning, and it produced RLM-27B.

In the same RLM harness, RLM-27B averaged 58.5 across the eight benchmarks against 46.1 for the untrained Qwen3.8-27B. The average weights each benchmark equally, and each benchmark uses its own scoring, so the abstract's 26.9 percent is a relative gain (12.4 points on a base of 46.1), not 26.9 points. The score rose on all eight benchmarks, though on BrowseComp-Plus only from 64.8 to 64.9. The trained model also worked differently. It handed work to sub-calls in 97 percent of runs instead of 43 percent, took 6.5 root turns per run instead of 18.4, and read about 44,000 root input tokens per run instead of 263,000. Adding the paper's per-run averages for root and sub-calls, total tokens fell from about 721,000 to about 385,000, which matches the abstract's roughly 2 times fewer, although sub-call output tokens rose. For scale, the same base model with no harness averaged 30.5. Reinforcement learning on the synthetic tasks without the program hints also helped, reaching 52.9, but brought no noticeable efficiency gain.

These are the authors' own experiments, and what they establish is narrower than the framing of the post. Locally in-distribution is an explanation rather than a measurement: the paper defines it loosely as each call seeing input the model can reasonably handle, and supports it with two checks, that root programs on long tasks resemble successful training runs and that sub-call accuracy held up where per-item answers were available. The side-by-side runs in the appendix were chosen to illustrate the point and are not a random sample. The synthetic tasks were built with knowledge of each target benchmark, so RLM-27B shows that targeted practice on stand-ins can teach a model how to divide up known kinds of long tasks; the paper does not test kinds of task it did not prepare for. The headline efficiency comparison counts turns and tokens, not running time. Within those limits, the paper makes a concrete case that how the harness divides work is something to train separately from the model's raw ability, and on these long tasks that targeted training paid off. Zhang discussed the idea of harnesses as compositional generalizers in a Latent Space interview, and a theoretical paper by a different team reaches a related point by another route, arguing that giving each subtask an isolated context rules out a shortcut that breaks when the domain changes.

Terms in this piece · Glossary
  • Program — A set of instructions stored so that a computer can run it.
  • Large language model — A model trained on large amounts of data to process and generate language.
  • Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
  • Harness engineering — Designing the full environment and feedback system around an agent.
Read the source alphaxiv.org
More from MIT CSAIL
Recommended reads
Comments

Checking sign-in…

Loading comments…