The inference bottleneck turned out to be the network

Z.ai reports that changing the network connecting its GPUs improved a GLM-5.1 serving cluster without changing the GPUs, applications or software stack. The ZCube design, which Z.ai deployed with Harnets.AI and Tsinghua University, removes an upper switch layer and uses complementary connection patterns to spread traffic that would otherwise concentrate on particular links. That traffic includes cached model state transferred between machines processing prompts and machines generating responses. The company explains the mechanism in its ZCube article.

In the reported before-and-after deployment comparison, average GPU inference throughput increased by about 15%, while the 99th-percentile wait for the first generated text fell by 40.6%. These workload-specific results show how network congestion can leave existing computing capacity underused.