Excited to collab with @ashxhart on this!
Why does prefill/decode disaggregation work on DGX Spark + Mac ?
DGX Spark: ~350 TFLOPS@fp4, 128GB@273GB/s
M5 Max MacBook Pro: ~70 TFLOPS@fp4, 128GB@614GB/s
Prefill is compute-bound. Prefill on Spark runs ~5x faster than M5 Max.
Decode is memory-bound. Decode on M5 Max runs ~2x faster than Spark.
Simple idea: run prefill on Spark, decode on M5 Max. That's faster than running on one device alone, since each phase runs on the best device for that phase.
This requires sending the KV cache from the Spark to the Mac. With a 10 Gigabit connection, we can overlap fully computation and communication of the KV cache. This doesn't require low latency, and works just fine over TCP/IP.
RDMA on the other hand enables low latency networking (~single digit microseconds), which is the unlock for scaling up decode with ~equal memory bw devices e.g. DGX Spark + M4 Pro / M5 Pro.
Low-latency is also important for DGX Spark + RTX or Mac + RTX. Small pool of very high speed memory + large pool of slower memory is the ideal local hardware architecture. Idea is to put the attention part of the model on the faster small memory pool, and the MoE part on the…
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Why it matters
Disaggregating prefill and decode across mismatched local machines is a way to get more out of hardware people already have rather than buying one bigger box.
Read the full post on X ↗