Inside the Together AI kernels team

Moving data around a GPU can be a performance bottleneck even when the chip has abundant computing power. Together AI's profile of its kernels team explains the role of kernels: software routines that arrange the hardware's calculations and data movement.

The team's research lineage includes FlashAttention. Its original paper's abstract describes dividing attention calculations into blocks to reduce transfers between the GPU's main memory and faster memory on the chip. This explains why software designed around memory behavior can improve model execution without simply buying a faster processor.

“We were prepping some blogs and code release for Tuesday morning. But then we saw Karpathy tweet about it Monday at 7 p.m. — so then we knew it was something people were paying attention to.”

Dan Fu

“Previous work on sparsity and low-rank methods showed theoretical speedup but only 10% real performance gains. The FlashAttention team took a different approach: understanding actual GPU memory movement and compute patterns. By applying classic database systems principles (unglamorous stuff about data locality and memory hierarchies) to attention, they achieved 2–3x speedups.”

Together AI

“By building abstractions around tensor cores, ThunderKittens reduces what was once 1,000+ lines of CUDA code to 100–200 lines.”

Together AI

“Within one week of hardware access, we had some of the fastest FP4 and FP8 GEMM kernels available for Blackwell, with up to 2x speedups over cuBLAS on H100s.”

Together AI

“The result: 77ms on Llama-3.2-1B, a 3.6x performance improvement and 7.2x better unit economics compared to their prior deployment.”

Together AI