The Base Model Is Dead — Varun Singh, Arcee AI

Arcee AI's pre-training lead argues that a model's early training should prepare it for the tasks it will later learn through rewards. In the presentation on base models, the speaker points to published training recipes that put question-and-answer chat data and reasoning or agent examples into the early training mix, so the model sees the shape of those tasks before reinforcement learning begins. The argument is that the initial model should hold task-relevant skills as well as general knowledge: as the speaker summarizes one paper, the model needs some exposure to the basic skills it must later combine during reinforcement learning. The title's claim that the base model is dead is therefore a thesis about its changing role; the proposed process still relies on supervised learning.

“web text, which used to make up like up to 85% of the train data in GPT uh 3, is now all the way down at 15%”

Varun Singh

“RL was mostly just a cherry on top, um shaping the, you know, flavor of the interactions more than conferring extra um knowledge or quality onto the base model itself”

Varun Singh

“it makes sense to view supervised learning as a way specifically to prepare the model for to build useful representations for for RL”

Varun Singh

“base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors”

Varun Singh