Split prefill and decode across a DGX Spark and a Mac
- Source
- x.com
- Date
Excited to collab with @ashxhart on this! Why does prefill/decode disaggregation work on DGX Spark + Mac ? DGX Spark: ~350 TFLOPS@fp4, 128GB@273GB/s M5 Max MacBook Pro: ~70 TFLOPS@fp4, 128GB@614GB/s Prefill is compute-bound. Prefill on Spark runs ~5x faster than M5 Max. Decode is memory-bound. Decode on M5 Max runs ~2x faster than Spark. Simple idea: run prefill on Spark, decode on M5 Max. That's faster than running on one device alone, since each phase runs on the best device for that phase. This requires sending the KV cache from the Spark to the Mac. With a 10 Gigabit connection, we can overlap fully computation and communication of the KV cache. This doesn't require low latency, and works just fine over TCP/IP. RDMA on the other hand enables low latency networking (~single digit microseconds), which is the unlock for scaling up decode with ~equal memory bw devices e.g. DGX Spark + M4 Pro / M5 Pro. Low-latency is also important for DGX Spark + RTX or Mac + RTX. Small pool of very high speed memory + large pool of slower memory is the ideal local hardware architecture. Idea is to put the attention part of the model on the faster small memory pool, and the MoE part on the…
Context
Alex Cheema argues that a DGX Spark and an M5 Max MacBook Pro working together can run a model faster than either alone, by giving each machine the phase it handles better. Prefill, the pass over the prompt, is compute-bound: he lists the Spark at about 350 TFLOPS at FP4 against about 70 for the M5 Max, and says prefill runs about 5x faster on the Spark. Decode, which generates the response, is memory-bound: both have 128GB, but the Mac's memory runs at 614GB/s against the Spark's 273GB/s, and he says decode runs about 2x faster on the Mac.
The handoff means sending the , the model's stored state for the prompt, from the Spark to the Mac. Cheema says a 10 gigabit link over ordinary TCP/IP lets that transfer overlap fully with computation, so this split does not need low latency. He saves RDMA, with latency in single-digit microseconds, for other setups, such as scaling decode across devices with similar memory bandwidth. The post gives per-device figures and ratios but no measured end-to-end result for the combined setup.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
- postIs the Rush to Build New LLM Inference Engines Fragmenting the Ecosystem?
- postBig model. 2.4T params, 95B active. 4.89TB. Surprised they didn't release a 4-bi
post@Jason @Lons @eisokant @ape Correction: Should actually be closer to 50 tok/sec
postApple markets a four-Mac cluster for trillion-parameter local inference
Checking sign-in…
Loading comments…


