Vibeleaderboard
← All Intel
Intel / article

Last year we built megakernels. This year we deleted them: The road to Together Compiler

Source
together.ai
Date

Context

Together AI's research team now says it no longer needs the record-setting megakernels it helped build last year, and its own measurements explain why (Together AI blog). A megakernel runs a model's entire forward pass, the computation that produces each new , as one long-lived GPU program instead of hundreds of separate ones, to avoid launch delays and keep the GPU's processors busy. On Llama 1B running on an H100, the team's megakernel reached 990 tokens per second against 669 for PyTorch's standard compiler. Getting it into production took months and yielded support for one model, at one batch size, in one numeric precision, for one customer.

The team, in a post by Dylan Lim, Reyna Abhyankar and Simran Arora, tested where that speed came from by starting from the standard compiler's output and adding megakernel benefits one at a time. CUDA Graphs, an NVIDIA feature that records a sequence of GPU launches and replays it with less overhead, raised throughput to 864 tokens per second. Programmatic Dependent Launch, which lets the next GPU program start before the previous one finishes, raised it to 953, and placing its wait instruction just before the first read of the previous result let the next step preload weights, reaching 975, or 98% of the megakernel, without writing a single kernel. The single-program design also had costs: every operation must share one launch configuration, which would make the heavier matrix multiplications 1.2 to 1.5 times slower on Hopper GPUs, and forcing Kimi K3's differently parallelized operations into one configuration cost 5 to 19%.

That finding pushed the team away from hand-built fusion toward coding agents that rewrite the forward pass's source code and measure the result. The harder lesson was about what to give those agents. Measurements on synthetic inputs misled them: one -written kernel ran in 20 microseconds on random data and 62 milliseconds on real inputs full of exact zeros, so the system now measures inside real forward passes on real data. An expert-curated knowledge hub made results 25 to 31% worse by anchoring agents to known patterns, and simply telling each agent to make the forward pass as fast as possible beat hand-written strategies. Together reports that the resulting system, Together Compiler, beats its old megakernel by up to 1.7 times and reaches 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 on DeepSeek v4.1-Flash, in under 24 hours on at most two nodes. Those are the company's own figures for a system offered by early-access request, and the post supplies the reasoning behind the compiler Together announced on November 4.

Key takeaways · AI-distilled
  • Together AI reports that default Torch Inductor kernels plus CUDA Graphs and fine-grained Programmatic Dependent Launch reached 975 TPS on Llama 1B on H100, 98% of its hand-written megakernel's 990 TPS, without writing a kernel. Inductor alone gave 669 TPS.
  • Inductor's kernel benchmarks misled its choices: clearing L2 cache made isolated timings 39-58% slower than in a live forward pass, and all-zero inputs ran 2.1x faster than random IDs. On one GEMM it picked a Triton kernel 15% slower than cuBLAS in real .
  • Megakernels force one launch shape on every operation. On H100, tensor-core GEMMs forced into the megakernel's 20-warp layout ran 1.2-1.5x slower than at 12 warps, and Kimi K3's KDA and MLA layers need tensor-core modes that PTX forbids mixing in a single kernel.
  • Synthetic inputs misled agents too: an agent-written top-k kernel ran in 20 microseconds on random data but 62 ms on real inputs, where ReLU zeros piled candidates into one histogram bin and triggered quadratic work.
  • Removing scaffolding helped the agents. On a MiniMax-M3 layer, a curated knowledge base cut ensemble speedup from 5.85x to 4.75x, a plain 'make it fast' prompt beat six expert strategies, and frequent message-board communication reached 7.72x versus 4.82x with none.
Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Read the source www.together.ai
Recommended reads
Comments

Checking sign-in…

Loading comments…