Intel / article

Together Compiler: Continuous Compilation for AI inference

Source: together.ai
Date: Published Nov 4, 2026

Context

Together Compiler is a system from Together AI, which hosts open models as a paid service, that uses AI agents to keep retuning how those models run as traffic, models and hardware change. Its announcement argues that hand-tuning no longer scales: Together sees prompt lengths shift by country and working hours, and reports an 80% rise in prompt length about two weeks after a new model's release as users move from chat to workloads. Each model, chip and traffic pattern can call for different code, and the company's answer is to let agents search for those optimizations continuously.

The bottleneck Together set out to remove is the time each experiment takes. Agent systems that tune serving software such as vLLM or SGLang usually restart the engine to test each change, and Together's own table puts that restart at 467 seconds, mostly spent initializing workers, loading weights, warming up kernels and autotuning. Together Compiler instead keeps a "persistent engine" running and hot-reloads each candidate forward pass, the code that computes one step of the model, which the table puts at 9.0 seconds to be ready to serve. Several agents propose ideas in parallel and share them through a message board. Some engines load only a few model layers for fast, steady timings; others load many layers to check whether a gain holds up on realistic input data, which matters for models whose speed depends on which experts the actual inputs reach. A separate cost model picks how to split the model across GPUs. Together credits this design with 252 times the experiment throughput of baseline agent-based optimizers.

The reported results are large but come with conditions. Together says that in under 24 hours the system reached up to 2.6 times the tokens per second of the best published open-source engine configurations on SemiAnalysis's AgentX benchmark, across models including Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, and 1.7 times the speed of open-source megakernels on their standard Llama 1B single-request test. Its claimed low-latency records, 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 on DeepSeek v4.1-Flash, used parallelism settings chosen by its own cost model rather than the benchmark's configuration, and a footnote says the Trainium2 runs used a synthetic dataset because its implementation could not handle AgentX's higher concurrency there. These are vendor numbers, not independent tests. The more durable claim is the method: if engine restarts dominate the cost of agent-driven tuning, removing them is what lets the search keep pace with traffic that keeps changing.

Key takeaways · AI-distilled

Together says engine restarts are the bottleneck for agent-driven tuning: a standard engine takes about 467 seconds to start serving, versus 9 seconds for its persistent engine, which hot-reloads each candidate forward pass instead of relaunching.
Benchmarking kernels on random inputs misleads for MoE and sparse , because the input distribution changes which optimization wins. Together tiers its engines: fast ones load a few layers for quick timing, slower ones load many layers to confirm gains on real data.
Together reports prompt lengths rise about 80% within two weeks of a model release as users move from chat to agentic use, and one country's working-hours prompts run 1.4x longer than another's (97k vs 70k), which is why it argues each deployment needs its own tuning.
To choose parallelism (TP, EP, DCP), Together uses a cost model that combines hardware specs, microbenchmarks and measured forward passes, since specs alone miss traffic effects such as MoE expert imbalance. Its batch-size-1 records, like 1111 TPS on MiniMax M3.1-Flash, used it.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

Read the source www.together.ai