How we make Together AI easy for coding agents to use
- Source
- together.ai
- Date

Context
Together AI, which sells access to AI models and GPU clusters, decided to find out how well coding agents could actually use its platform by writing down the jobs developers do there, such as running a batch inference job or reserving a cluster, and handing each one to an as a prompt. Every task runs in three setups: the agent alone with its ordinary tools, the agent with access to Together's documentation through an MCP server, and the agent given Together's package, a set of written instructions and examples meant for agents rather than people. Three models attempt each task twice in each setup, in an isolated , with a proxy recording which API endpoints the agent called and what came back. An then reads the prompt and the recorded session and marks the run pass, incomplete or fail, which the post describes as pointing at where to look rather than as a diagnosis.
The first case study shows why the recording matters more than the verdict. One task asks an agent to list available GPU cluster regions and instance types, print the cheapest option and show the reservation command without running it. Last summer Sonnet 4.6 passed 4 of 15 attempts even with the skill package loaded. The transcripts showed the model reading a sample output in Together's CLI reference as though it were the live inventory, and never checking prices, because the skill file pointed at a marketing page instead of the pricing table. In the same period Opus 4.8 passed 13 of 14 attempts on the same setup by noticing the missing prices and hunting them down in the docs, which is the post's sharpest point: a more capable model papered over a defect in the product, and testing only with that model would have hidden it. Adding two sentences that name the live regions endpoint and the pricing table took the task to a pass every time for all three models tested with the skill config. A separate run exposed a genuine bug, where an API error message told the agent to call a route that does not exist, sending it into a dead end a human would have hit too.
The second case study is Together marking its own homework and reporting a failure. The company merged 14 separate product skills into one and expected an improvement, then measured the new version against the 90% pass rate its 14 skills had reached on September 30. The unified skill passed 54 of 72 runs, left 18 incomplete against 7 before, and cost more than twice as much per run. The transcripts pointed to three problems: a router of about 14,000 characters that loaded on every task and told agents to read each guide in full, so runs pulled in 57% more skill content than the separate skills had; code samples that used models no longer served; and bugs in the example code. Cutting the router below 6,000 characters, giving each guide a menu so agents open only what they need, moving samples to live models and fixing the example code brought it to 70 of 72 runs, 7 points above the 14 skills, at 20% less time per run and the original cost.
The four-month trend is the weakest part of the evidence, and the post says so. On June 15, running Opus 4.8, the baseline passed 75% of runs while the documentation MCP server reached 67% and skills 75%, so the guidance was not helping. By September 30, across three models, the baseline had reached 82%, the MCP server 96% and skills 90%, with the unified skill at 97% on October 8. Together states that this is not a controlled comparison, since the snapshots used different models and tasks, says better models and its own CLI and SDK work probably explain the 7-point baseline rise, and credits the larger gain in the guided setups to the documentation and skill fixes its evals turned up. The reading that survives this caution is the narrower one: when agents fail on a developer platform, the fault is often in the documentation, error messages and agent instructions, and that is only visible if the failures are recorded and read. Together says it will open-source the package it uses to run these evals, along with a follow-up post, so other platforms can measure the same way.
- Together AI runs each agent eval task three ways: a baseline with default tools only, one with docs through its MCP server, and one with its skill package. Each runs on three models with two attempts per task, and an LLM judge grades sessions pass, incomplete or fail.
- Sonnet 4.6 passed a GPU-pricing task 4 of 15 times: it took unlabeled sample CLI output as live inventory, and a skill linked a marketing page, not the pricing table. Opus 4.8 worked around the gap, so Together says testing weaker models exposes problems.
- A two-sentence skill addition pointing agents to the live regions API and the pricing table took that GPU-cluster task to a 100% pass rate for every model tested with skills enabled.
- Transcripts also exposed an API bug: an error message told agents to call /v1/clusters/regions, a route that does not exist. Together changed the message to point at /v1/compute/regions and fixed the routing bug behind a related 404.
- Merging 14 skills into one first cut the pass rate to 75% at over twice the cost per run, partly because a 14,000-character router loaded on every task. Trimming it under 6,000 characters and fixing stale models and buggy samples raised it to 70 of 72 runs.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
- sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
Checking sign-in…
Loading comments…

