Long Context Latency Scaling Gpt Vs Claude

Epoch AI researcher Jason Li timed how long four frontier models take to return the first token of a reply as prompts grew from about 50,000 to roughly 900,000 tokens, and found that the two model families slow down in different shapes (epoch.ai). For OpenAI's GPT-5.6 Terra and Sol, each extra block of context adds more waiting time than the block before it, a curve consistent with a significant quadratic component. For Anthropic's Claude Sonnet 5 the added delay stays close to constant, so latency rises in a nearly straight line, and Claude Opus 5 is noisier but still consistent with linear scaling. Li reads the gap as a sign that the two labs made very different architectural choices about how their models process long prompts.

Time to first token is a useful window because it is dominated, at long lengths, by prefill: the phase in which the model reads the entire prompt before it writes anything. In a classic transformer every layer uses full attention, where each token looks back at every earlier token, so the work of reading a prompt grows with the square of its length. Many open models now cut that cost. Hybrids such as Kimi K3 and Qwen3.5 interleave cheaper linear-attention layers with full attention at a 3:1 ratio, others swap in sparse attention, and OpenAI's open gpt-oss models used sliding-window attention, which looks only at a fixed stretch of recent tokens. Claude's near-linear curve suggests that full attention contributes much less to its long-context processing than it does for GPT-5.6; the post infers this from timing, not from disclosed model details.

API timings are noisy, partly from queueing and routing, so Li fit the data three ways with different assumptions about that noise. All three recovered similar upward curvature for both GPT models. Sonnet stayed at or near zero curvature, while the Opus result is described as inconclusive rather than proven linear. In concrete terms, adding 10,000 tokens costs GPT-5.6 Terra about 0.071 seconds at a 100,000-token context and about 0.216 seconds at one million, while Sonnet 5 adds about 0.129 seconds at both. Li connects this to product defaults: Claude 5 models in Claude Code support a one-million-token window by default, while Codex autocompacts before 272,000 tokens even though GPT-5.6 supports 1.05 million, and OpenAI charges double the input price above 272,000 tokens where Anthropic keeps one price regardless of length.

The post also extrapolates the fitted curves to ten million tokens, explicitly as a stress test rather than a forecast, since the measurements stop near one million. On those curves the GPT models' marginal delay per added token would be roughly 7 to 8 times what it is at one million, which Li argues would push long-running agents on quadratic models toward external memory and aggressive compaction, while near-linear models could keep more history in context. An appendix extends the measurements to newer OpenAI models: GPT-6 Astra still shows a significant quadratic component, and GPT-6.1 Sol has a fitted quadratic coefficient 59% lower than GPT-6 Sol, starting slower at about 50,000 tokens but finishing faster at about 850,000.