Scaling Compute on Context — Jack Morris, Engram

“Train a model directly on ten thousand financial reports and you can drive the loss to 0.00001. It knows the documents perfectly. Then you generate from it and it collapses.”

“Models are superb on Wikipedia, arXiv, and GitHub, and know nothing about your emails, your meetings, or your company.”

“Every approach shares a ceiling: you define a dataset, you train, and unless the model is underparameterized it eventually absorbs everything you made. A synthetic data wall, with none of pretraining's scaling behavior.”

“The property he wants is the one that made AlphaGo work, where getting better makes the training questions harder, so that adding compute keeps buying depth instead of flattening out.”