grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
If you're evaluating whether to trust LLMs to generate performance-critical GPU code, ParallelKernelBench gives you a groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → reality check: frontier models solve fewer than a third of multi-GPU kernel workloads, yet occasionally produce kernels that outperform any public implementation.
Key quotes
“communication overhead can account for over 20% of inference latency”
Together AI
“In the zero-shot setting, the best model solves 28 of 87 problems, and only 22 of those solutions are faster than the PyTorch + NCCL baseline.”
Together AI
“Sampling three attempts improves the best result to 36 correct solutions and 27 faster-than-baseline solutions, but fast 1 @3 still tops out at 31%.”
Together AI
“Weaker models often fail to compile, but stronger reasoning models frequently produce kernels that compile and return incorrect results. The hard part is reasoning about rank coordination, data partitioning, and collective ordering.”
Together AI
“Gemini 3 Pro improved from 24 correct solutions in the single-shot setting to 35 out of 87, with 26 kernels beating the PyTorch + NCCL baseline.”
videoCan LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AIAI Engineer
articleDataKernelBench: Can LLMs Optimize Database Queries on GPUs?Gokul Karthik Kumar, Yotam Perlitz, Corey Lammie, Andrea Giovannini, Katja Hose