Polymorphic Inference: self-tuning inference for…

Context

Polymorphic is the label Together AI, which runs open models as a hosted service, gives to a change in how it thinks inference should be run. In this post by research leads Simran Arora and Dan Fu, the old approach, which they call static inference, picks one tuned configuration for a model, chip and traffic pattern and leaves it in place. Their argument is that none of those stay fixed: new open models arrive quickly and with different architectures, customers' request patterns keep changing, and hardware generations turn over fast. A configuration or kernel (a piece of GPU code for one operation) tuned for one setting may be wrong for the next, and kernels tested on random inputs can behave differently on real traffic, especially for models.

The post includes an admission about Together's own earlier work. Nearly two years ago the team wrote about megakernels, single fused GPU programs hand-built around one fixed case such as a 1B Llama model serving one request at a time. After putting some of them into production, the authors write, they concluded that such fusion is the wrong design goal when architectures and batch sizes vary. Their replacement is to have AI agents do the tuning continuously, which they call Continuous Compilation, carried out by a system named Together Compiler that treats optimization as a search by many agents testing changes against a running engine.

Much of the post is plan rather than product. Together says Together Compiler already runs on some of its most popular serverless endpoints and has found gains over code that was already heavily optimized, and that customers can request a limited beta; a self-serve version for dedicated endpoints is described as coming soon. The other items, that adapts per request and per token, a router building on its earlier ThunderAgent scheduling work, post-training and workload-aware , are presented as a preview of its research agenda, and the post reports no results for them. Market claims such as 80% of OpenRouter traffic going to models released in the last three months are stated without a source. Readers should take the post as Together's case for where inference engineering is heading, with its supporting measurements in the separate Together Compiler announcement.

Key takeaways · AI-distilled

Together cites OpenRouter data that 80% of traffic goes to models released in the last three months, and notes the ten most-used open models all use different architectures, which it argues makes per-model hand tuning unworkable.
Together says running megakernels in production taught it that fusing a model into one kernel built for a single static shape, such as Llama 1B at batch size 1, is the wrong goal once MoE architectures and dynamic batches larger than one are involved.
Published vLLM, SGLang and InferenceX recipes launch with a fixed tuned config and a fixed speculative decoding draft model and lookahead. Together argues both should adapt per domain, request and traffic load, and lists shape-shifting speculative decoding as upcoming work.
Together Compiler recasts inference optimization from compiler translation to agentic search, pairing a swarm of agents with a persistent engine. Together says it runs on popular serverless endpoints and is in limited beta, with self-serve tuning for dedicated endpoints planned.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
model routing — Sending each request to a model chosen by the difficulty of the task, rather than using one model for everything.