Interconnects AI: Reading Today's Open-Closed Performance Gap
Source
Nathan Lambert
Author
Nathan Lambert
Date
Key takeaways · AI-distilled
What benchmarks measure rotates every 12 to 18 months: chat and math, then reasoning and complex code, now agentic knowledge work in law, accounting, and healthcare. A gap number mostly tells you who caught up to the previous cycle.
The data market behaves like chip fabs. US frontier labs pay astronomical sums for new RL environments and datasets; fast-followers buy the same assets later at a steep discount. Catch-up is purchased, not only distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition →.
Open models are weakest in robustness rather than peak capability. Long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → handling is the tell: you reset an open-model AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s context far more often than with Claude or Codex.
Frontier revenue assumes coding and terminal tasks stay hard. If those saturate and a cheap open model does the job, enterprise spend rests on relationships, inertia, and product, not on models being leaps ahead.
Hard tasks are getting harder to hill-climb because the data is private. Code had all of GitHub; accounting and legal workflows have no comparable public corpus.
Terms in this piece · Glossary
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
If you're deciding which models to build on, this reframes the open-vs-closed 'gap' as benchmark-dependent and explains why fast-follower labs stay close — useful for judging when open weightsA model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.Full definition → are good enough for your use case.
Key quotes
“In a new era of rapid post-training improvements, I",”
