Tracing Olmo 3 capabilities back to pretraining data categories

A team led by Georgia Tech researchers asked where a language model's social reasoning, its ability to read people's beliefs, feelings and intentions, comes from in its training data. Their paper, published at COLM 2026, answers that for the 7-billion-parameter Olmo 3 from Ai2. The work depended on Ai2 releasing the training corpus as well as the model; the authors note that a model published with weights alone cannot be traced back to individual documents this way.

The tool is training-data attribution, which estimates how much a given training document pushed the model toward its answer on a test question by comparing the change that document would cause during training with the change the question calls for. Scores for single documents are too noisy to trust, so the team sampled about 5.7 million documents from the web portion of the roughly 1.26 billion unique documents in Olmo 3's de-duplicated training mix, labeled each with one of 24 topics and one of 24 formats, and averaged influence within each of the 576 combinations. They repeated this for four benchmarks chosen to separate subject from skill: SocialIQA for social reasoning, ARC-Challenge for science reasoning, and the social-science and STEM sections of MMLU for factual knowledge.

SocialIQA was the outlier. Its strongest support came from interpersonal and narrative text, with customer-support exchanges as the leading format, and literature helped it while counting against the other three tests. Those three drew mainly on documentation and other expository writing, and their profiles resembled one another closely while SocialIQA's barely matched any of them. As Ai2's summary puts it, social-science knowledge looked more like STEM than like social reasoning.

Attribution shows association, not necessity, so to test whether the high-influence documents are actually needed the team had the model unlearn the 200 most influential documents within a topic and compared the damage with forgetting 1,000 random documents from the same topic. For SocialIQA, targeted forgetting hurt clearly more. For the other benchmarks the effect was weaker, not significant, or for ARC-Challenge slightly reversed, so the method did not reliably find essential documents across the board. On a wider set of nine social tests covering theory of mind, moral judgment, bias and pragmatics, seven shared a profile tied to more formal, documentary text, and SocialIQA sat outside it. The authors therefore present the result as mapping one region of social ability rather than social reasoning as a whole, and Ai2 cautions that it is not a recipe such as adding more literature to get better social reasoning.