A study of sequence weighting at scale

Giving a training example more weight does not have the same effect at every model size. In Jane Street's experiments, researchers assigned random weights to text sequences, trained language models, then measured how much prediction error fell on those same sequences. This isolates the influence of weighting from differences between data sources.

The relationship often strengthened as models grew from small to medium size, then weakened at larger sizes. The researchers hypothesize that smaller models mainly learn shared patterns, intermediate models prioritize particular examples, and larger models can fit more of those patterns regardless of weight. The finding makes small-model experiments an uncertain guide to weighting data for a much larger training run.