Two scores for the same model can answer different questions when the surrounding software changes. ARC Prize's testing policy separates a provider-neutral ARC-AGI-3 setup from one using provider-designed context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → management. Features such as retaining reasoning state between requests can change how an interactive run proceeds.
The policy requires those conditions to be labeled separately. It also distinguishes independently verified results from community submissions, which are not verified by default. Before treating a leaderboard number as a model comparison, check the benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → version, evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → setup and verification status. A shared model name or leaderboard page does not make those conditions interchangeable.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
determinism — Whether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.
