ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

Writing a correct GPU program and writing a faster one are separate hurdles. Together AI's ParallelKernelBench tests both across 87 tasks involving communication between GPUs within one machine. Models replace a standard implementation with code that coordinates computation and data transfers directly. The benchmark report measures correctness separately from beating the reference implementation.

With one attempt per task, the best evaluated model produced 28 correct solutions, of which 22 were faster than the reference. Selecting from three attempts raised those counts to 36 and 27. Thus the strongest three-attempt result improved performance on about 31% of all tasks. The benchmark exposes the gap between generating plausible distributed code and delivering a verified optimization.

“communication overhead can account for over 20% of inference latency”

Together AI

“In the zero-shot setting, the best model solves 28 of 87 problems, and only 22 of those solutions are faster than the PyTorch + NCCL baseline.”

Together AI

“Sampling three attempts improves the best result to 36 correct solutions and 27 faster-than-baseline solutions, but fast 1 @3 still tops out at 31%.”

Together AI

“Weaker models often fail to compile, but stronger reasoning models frequently produce kernels that compile and return incorrect results. The hard part is reasoning about rank coordination, data partitioning, and collective ordering.”

Together AI

“Gemini 3 Pro improved from 24 correct solutions in the single-shot setting to 35 out of 87, with 26 kernels beating the PyTorch + NCCL baseline.”

Together AI