Vibeleaderboard
← All Intel
Intel / article

Making open models work for long-horizon agentic workloads

Source
together.ai
Date

Context

A coding on an open model stays fast through a long session only if two separate pieces of software protect one fragile shortcut, according to a joint post from Together AI, which runs inference for open models, and Ollama, which connects coding harnesses such as Claude Code and Codex to open models (Together AI blog). On every step, the , the program that drives the agent, resends the whole session: , tool definitions, every file read and command run. The model usually replies with a tool call a few dozen tokens long, so nearly all the work is reading input. Prefix caching lets the inference engine reuse its work on the part of the prompt it already processed on an earlier turn. In the authors' idealized arithmetic, where every turn adds the same amount, a hundred turns cost 5,050 units of prompt processing without caching and 100 with it.

The shortcut breaks easily because the cache matches only from the start of the prompt up to the first token that differs, and everything after that point is processed again. That makes the harness responsible for where changes land. A timestamp, git status or working directory in the system prompt turns every turn into a full re-read. So does adding a newly discovered tool to the tool list near the top instead of at the end of the conversation, although the authors note that some models' chat templates reject a tool definition placed mid-conversation. Ollama hit a subtler case with GLM-5.3-Flash, whose chat template writes the reasoning-effort setting into the first system block, so raising effort partway through a task forces the whole session to be reprocessed. Their advice is to pick an effort level at the start, or begin a fresh session at the higher setting.

The engine has to keep the cache alive and send each turn back to it. A session's cache sits on one machine, so Together routes the next turn to the node holding that prefix, either by a key the client sends with each request or by matching the prompt against what each node has cached. It moves an idle session's cache to CPU memory while the agent waits for tests to run, and it reads prompts and writes answers on separate GPU clusters so a wave of long agent prompts does not slow for other users. Even then, the authors write, a warm node can be so busy that taking a cache miss on an idle node is the better deal, so a perfectly stable prompt does not guarantee a hit. Vercel's AI Gateway tackles the same routing problem one level up with session affinity, which sends each turn to the provider that served the one before.

The conclusion for anyone pairing a harness with an open model is that a slow or expensive long session is often a cache failure rather than a slow model, and either side can cause it. The post gives no measured hit rates or cost figures, so it explains where caches break without showing how often they do in practice.

Key takeaways · AI-distilled
  • A prefix cache matches only up to the first changed token. In the authors' idealized math, ten equal turns cost 55 units of prompt work without caching versus 10 with it, and 5,050 versus 100 at a hundred turns: uncached work grows with the square of session length.
  • Anything that changes near the top of the prompt, such as a timestamp, git status or working directory in the system prompt, forces a full re-read each turn. Per-session values like a user's name also stop new sessions from reusing an already cached shared setup.
  • In GLM-5.3-Flash, the chat template writes reasoning effort into the first system block, so raising effort mid-session reprocesses the whole transcript. The authors suggest choosing effort up front, or starting a fresh session or compacting before changing it.
  • Truncated tool output should say what was cut. In the post's example, a file read returned lines 1 to 100 of 640 silently, hiding the bug at line 420; one line stating the range lets the model search for and read the right section instead of editing code it never saw.
  • On the engine side, a turn routed to a different node recomputes the whole prefix. Together routes by a client cache key or longest matching prefix, offloads idle caches to CPU memory during tool waits, and runs prefill and decode on separate clusters linked over RDMA.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
  • streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
Read the source www.together.ai
Recommended reads
Comments

Checking sign-in…

Loading comments…