Vibeleaderboard
← All Intel
Intel / post

Cheap agents are closing the CUDA portability gap

Source
x.com
Date
Alex Cheema@alexocheema

A year ago the consensus was everyone would be using one or two frontier models, running entirely on NVIDIA GPUs. The reality now is we have hundreds of models each with different cost/speed/intelligence tradeoffs for different tasks, running on dozens of different local / data center chips (even older hardware). Ironically the frontier models that were trained on NVIDIA GPUs have made it faster and easier to port kernels from CUDA to other hardware targets (AMD, Apple Silicon, Intel, ...). There are people with no experience writing kernels using relatively cheap AI agents to implement efficient kernels for Apple Silicon, synthesizing all the tricks from existing CUDA implementations that were meticulously hand written by experts. Funnily enough, since the performance of these kernels can be quickly and objectively evaluated by an agent by actually running the kernel on the hardware, it's an unreasonably tractable task for agents. They can continuously improve them in a fairly simple autoresearch loop. Kernel interoperability is solved ("The unreasonable effectiveness of AI agents"). The next continuation of this trend is to have millions of small, specialized models for…

Read the full post on X

Context

Alex Cheema's argument rests on one property of GPU kernels, the small low-level programs that do a 's heavy arithmetic on a specific chip: whether a new kernel is faster can be checked by running it. Because an gets an objective score after each attempt, Cheema argues it can keep improving kernels in a simple loop, which is why people without kernel experience are porting tricks from hand-tuned NVIDIA CUDA code to Apple Silicon. The post offers this as opinion, with no benchmarks, and goes as far as calling kernel interoperability solved.

Published evidence supports the loop but not the word solved. In K-Search translation work covered on Intel, researchers from UC Berkeley and IBM Research describe a system that generates candidate kernels, compiles and benchmarks them on the target Apple hardware, and feeds the measurements back into the search (BAIR blog). Their kernel went from 0.26 times to 0.97 times the speed of Apple's native MLX version, and a Mamba prefill kernel ran roughly 20 times faster than the existing mlx-lm implementation because the search found a parallel scan it lacked.

Those gains depended on more than measurement. The researchers gave the model curated tables mapping CUDA concepts to Apple's equivalents, including hardware limits such as smaller on-chip shared memory, and they tested only two kernels. They say they do not yet know how far the approach generalizes. That makes Cheema's direction plausible for kernels with a clear speed target and access to the real hardware, while the claim that portability is finished remains his projection.

Terms in this piece · Glossary
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
More from Alex Cheema
Recommended reads
Comments

Checking sign-in…

Loading comments…