Intel / article

From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon

Source: bair.berkeley.edu
Date: Published Jul 29, 2026

Context

IBM researchers adapted K-Search, an AI system that writes and tests GPU routines, to Apple Silicon by supplying hardware-specific constraints and translations of existing optimization techniques. Generated candidates run on real hardware, and their measured performance guides further revisions. The study shows why useful hardware knowledge matters alongside code generation.

Its clearest gain came from processing a prompt with mamba-370m in half precision on a 64GB M1 Max. The optimized implementation achieved roughly twenty times the throughput of the tested community mlx-lm baseline on long inputs. It reorganized sequential state updates into a parallel computation. Generating one new text unit at a time offers much less parallel work, so that phase improved far less.

Key takeaways · AI-distilled

Handing an a CUDA kernel to port directly produces syntactically valid but architecturally wrong code; the translation layer instead supplies concept-mapping tables, e.g. __shared__'s 32KB Metal limit versus NVIDIA's 48KB memory.
On the kernel, giving the evolutionary search full architecture-specific jumped performance from 0.26x to 0.97x of Apple's own state-of-the-art attention kernel, independently rediscovering FlashAttention-2 techniques like online softmax and the exp2 trick.
The roughly 20x Mamba SSM prefill speedup over mlx-lm comes from exploiting that the state recurrence step is associative, letting it run as an O(log N) parallel scan instead of mlx-lm's O(N) -by-token processing; the gain shows only in prefill, not decode.
K-Search's search state is a scored decision tree, not a flat list: each candidate node carries an overall rating (0-10), a confidence (0-1), and separate impact scores for memory bandwidth, register pressure, and compute fit, with a stagnation window triggering a branch switch.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source bair.berkeley.edu