A reasoning modelA model trained to think — generating extended internal reasoning before answering — trading time and tokens for accuracy on hard problems.Full definition →'s benchmark score needs a budget beside it. In its early o1 evaluation, ARC Prize tested OpenAI's model with the same baseline setup used for other models and found that the extra reasoning came with substantially longer execution times. A score alone therefore left out an important part of the comparison.
The article argues for comparing accuracy with the computing effort spent reaching it. More attempts can improve a result without making the underlying method equally efficient. Its discussion of o1's internal training and search mechanisms includes explicit speculation; the evaluation establishes results for the tested setup, not those architectural explanations.
Terms in this piece · Glossary
reasoning model — A model trained to think — generating extended internal reasoning before answering — trading time and tokens for accuracy on hard problems.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
