TypeSafe AI argues that repeatedly choosing models by the same public benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → can make that benchmark part of development, even without training directly on its questions. Trying many settings and keeping the highest-scoring result lets the test influence what gets built. In Lies, Damned Lies, and Benchmarks, the company says that for its own model releases it will skip standard benchmark tables, publish new evaluations as dated snapshots that are retired once posted, and share its internal evaluations with caveats, cherry-picking and unfavorable evidence. The stated aim is to avoid hill-climbing a public score and to make the basis for its capability claims easier to inspect.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
