The End of TCP for AI Clusters — John Ousterhout, Stanford
Source
AI Engineer
Author
AI Engineer
Date
Key takeaways · AI-distilled
Agentic inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → has shrunk GPU compute phases to milliseconds, so the metadata exchange between rounds (checking a KV-cache entry, clearing a barrier) now costs about as much time as the computation itself, unlike the old gigabyte-gradient, throughput-bound era.
TCP's congestion control runs on the sender, but queues actually build at the receiver's last hop, so the sender learns about congestion only indirectly, via roughly one bit of signal several round trips late — a known flaw for 20 years that still makes TCP oscillate.
Homa, built at Stanford from Behnam Montazeri's dissertation, moves congestion control to the receiver: since it knows from the first packet how much data is coming, it paces delivery of the rest by issuing explicit grants.
Homa replaces TCP's byte-stream model with explicit messages, so short coordination messages (cache checks, barrier signals) can overtake long transfers instead of queuing behind them, reusing switches' existing priority queues rather than new hardware.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Explains why traditional sender-driven TCP congestion control now bottlenecks AI clusters as agentic inference compute phases shrink to milliseconds, and details Homa's receiver-driven approach as a structural fix.
