Together Compiler: Continual Compilation for AI Inference
Source
arxiv.org
Author
Reyna Abhyankar, Dylan Lim, Drew Wadsworth, Venkata Hasith Vattikuti, YuTong Julia Zhang, Jung Hwan Heo, Taeksang Kim, Kesavan Ramakrishnan, Evangelene Stanley, Xinyu Fang, Daniel Y. Fu, Simran Arora
Date
Why it matters
Shows how an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → hill-climbing loop can keep inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → deployments optimized as models and traffic shift, avoiding both low-fidelity kernel benchmarks and slow end-to-end engine launches.
Key takeaways · AI-distilled
To get fast and faithful measurements at once, TKCC adds a Tiered Persistent Engine that decouples engine launch overhead from forward-pass computation, plus a cost model that picks the parallelism configurations seeding optimization.
Together AI reports TKCC beats hand-written Megakernels on a Llama 1B workload by up to 1.7x, and the best open-source engines on Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek v4.1 Flash by up to 2.6x TPS on InferenceX.
At batch size 1 the paper claims state-of-the-art throughput of 1111 TPS on MiniMax M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash, and Together says TKCC already serves trillions of tokens across its production fleet.
Terms in this piece · Glossary
agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
