Are Open Models Catching Up?
- Source
- newsletter.semianalysis.com
- Author
- Evan Cloutier
- Date
Context
SemiAnalysis measures how quickly open models reach the first closed-model reference in each of three periods: early scaling, reasoning and agentic work. Its analysis reports shorter catch-up times across those periods. Reaching an era's initial reference does not establish parity with whatever model leads later.
The method uses a different curated set of benchmarks for each period. In the first period, for example, scores were normalized so the best result was 100 and then averaged with equal weight across four benchmarks. The resulting scores describe selected comparisons, rather than one continuous scale of intelligence. The authors also acknowledge that rankings can differ from their preferences for everyday work.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…