PCIe lanes, not slots or money, cap a build. At six cards the author swapped a Supermicro H12SSL-i motherboard for an ASUS ROMED8-2T purely for lanes, then hit the ceiling again until MCIO adapters freed up more.
Going multi-node was the real jump. llama.cpp RPC over Ethernet had too much latency, so the fix was used datacenter gear from eBay: a Mellanox SB7800 InfiniBand switch and ConnectX-6 cards, which then needed 3D-printed cooling shrouds.
The nastiest bug was power. Dell D2400E-S0 supplies would randomly cut power to a GPU with no relation to load and no way to reproduce it. Swapping to 2000W Super Flower units is what finally made the cluster predictable.
Software was harder than hardware. Getting vLLM plus Ray running across the cluster took more than 100 Docker configurations and was the point where the author nearly quit.
The economics only flip at volume: roughly $30k for GPUs and server components, bought before the memory price spike, which beats API pricing once you process billions of tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → a month — and keeps the data local.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
First-person account of scaling a personal GPU cluster from a single RTX 3090 to 32 GPUs over two years — the practical decisions, challenges, and lessons for anyone building serious home AI/ML infrastructure.
Key quotes
“Two years later I have 32 RTX 3090s across 4 servers, 768GB of VRAM total, connected with InfiniBand and running on solar power.”
“once you process billions of tokens a month, running your own hardware is cheaper than paying for an API, and your data stays with you.”
“One thing I'd recommend: clone your SSD right after you get your switch. Make the clone first, then experiment, so you can always roll back.”
“The question I get most: roughly 30k just for the GPUs and server components.”
