Gives the math for when self-hosting an open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → model beats buying API tokens, from weight memory through batching to price per token. Helps teams estimate the break-even point.
Key takeaways · AI-distilled
Cline's math puts Kimi K2.6 in FP4 at about 500 GB of weights, so an 8x B200 node (about 1.5 TB) leaves roughly 956 GB for KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition →. At $6 per GPU-hour that node costs about $35K a month, which the author calls the minimum entry ticket for self-hosting.
The post explains that decode is bandwidth-bound and prefill compute-bound. Batching shares weight reads across users until per-user KV reads dominate, and past the KV cache limit evictions trigger a collapse where time to first tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → jumps to tens of seconds.
In CoreWeave load tests on Cline's traffic, TP8 peaked at batch 16, but splitting each node into two TP4 replicas gave about 63% more throughput, at roughly $0.25 per million processed tokens versus $0.317 via API, since smaller tensor-parallel groups spend less time syncing.
Sizing the self-hosted fleet to the overnight traffic trough and spilling the rest to the API through LiteLLM saved Cline about 10% (roughly $19K a month). The author estimates time-of-day autoscaling could raise savings to 22-25%, which was not yet implemented.
The author warns that first-principles estimates miss real stacks by 2-10x (their napkin math was 4x off measured per-pass time) and advises against self-hosting below about $500K a year of API spend, since one inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → engineer costs $300-500K.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
