Interconnects AI: Reading Today's Open-Closed Performance Gap
- Source
- interconnects.ai
- Author
- Nathan Lambert
- Date
Context
A small gap in an overall score can conceal larger differences on particular tasks. Nathan Lambert's analysis of open and closed models argues that the relevant question is which capabilities the benchmark measures and how closely they match actual work.
The analysis connects this measurement problem to a changing training target: answering isolated questions differs from completing extended tasks with software and specialized tools. As those targets change, access to suitable training environments also matters. The argument gives model buyers a reason to examine their own workflows, while the prediction that open models will keep trailing remains a forecast.
- What benchmarks measure rotates every 12 to 18 months: chat and math, then reasoning and complex code, now agentic knowledge work in law, accounting, and healthcare. A gap number mostly tells you who caught up to the previous cycle.
- The data market behaves like chip fabs. US frontier labs pay astronomical sums for new RL environments and datasets; fast-followers buy the same assets later at a steep discount. Catch-up is purchased, not only .
- Open models are weakest in robustness rather than peak capability. Long- handling is the tell: you reset an open-model 's context far more often than with Claude or Codex.
- Frontier revenue assumes coding and terminal tasks stay hard. If those saturate and a cheap open model does the job, enterprise spend rests on relationships, inertia, and product, not on models being leaps ahead.
- Hard tasks are getting harder to hill-climb because the data is private. Code had all of GitHub; accounting and legal workflows have no comparable public corpus.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
“In a new era of rapid post-training improvements, I",”
Checking sign-in…
Loading comments…