The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference

In SemiAnalysis's agentic inference tests, Google's TPUv7 Ironwood chip is the cheapest or a competitive way to serve all four large open models tested, but NVIDIA GPUs running the same models at 4-bit precision still lead on two of them, GLM-5.2 and Kimi K3 (SemiAnalysis). SemiAnalysis, a chip research firm, runs AgentX, a benchmark that replays long coding-agent sessions with many turns, growing context and short-lived sub-agents, and compares how many tokens each system produces per dollar at a given speed for each user.

Precision decides much of the outcome. FP8 and FP4 store each model weight in 8 or 4 bits, and halving the bytes speeds up generating text for small batches, which is limited by memory. 4-bit can cost some accuracy when a model was not trained for it. TPUv7 has no native FP4 support, and SemiAnalysis expects FP4 to arrive with TPUv8 in the first quarter of 2027. On GLM-5.2, TPUv7 delivered 1.2 to 2.7 times the tokens per dollar of NVIDIA's B200 and B300 and AMD's MI355X at 8-bit, yet NVIDIA's GB300 and GB200 at 4-bit delivered 3.6 and 3.5 times TPUv7's figure at 100 tokens per second per user. On Kimi K3, TPUv7 was close to GB300 at launch, but after four NVIDIA software updates GB300 at 4-bit leads by 1.5 to 2.6 times. On Qwen3.5 397B, the first model Google submitted, TPUv7 at 8-bit scored 1.67 times better per dollar than GB300 at 4-bit. NVIDIA's counter submission beat it on generating tokens but, SemiAnalysis says, lost ground on processing the prompt, so TPUv7 still led end to end.

The broader argument is about software. NVIDIA's advantage has long rested on developer tools, and these TPU results came through open-source serving engines such as vLLM and SGLang, built with outside maintainers including Inferact and RadixArk. One technique, a megakernel, runs a whole generation step as one long kernel so the chip does not sit idle between hundreds of separate launches. Meta's Muse Spark Open, a model SemiAnalysis calls extremely GPU-optimized because its attention shapes leave a quarter of the TPU's matrix unit padded, still came out competitive on generation cost per dollar, though TPUv7 needs faster prompt processing to compete on end-to-end latency.

These are SemiAnalysis's comparisons, and the free portion of the article does not fully define the dollar cost behind them. Most figures are described as costs for external customers, the Muse Spark Open result uses what Meta would pay to buy TPUs, and Google's internal cost is mentioned separately. The rest of the article is paywalled, and SemiAnalysis's NVIDIA Rubin numbers are still preview submissions.