Xaira broke through its scaling wall by collecting roughly 30x more information per dataset. After that, adding parameters and compute started buying performance again. The fix was running new experiments, not building a bigger model.
CELLxGENE holds 168M cells, each a count over 20-30K genes. It tells you which genes move together, and correlation alone can't say which one drives which, so models trained on it nail cell types and fail at predicting gene edits.
CRISPR lets you turn one gene down at a time and watch what changes downstream. Run millions of those in parallel and you get causal data: A causes B, not just A and B co-vary. That is the X-Atlas dataset underneath X-Cell.
The budget shape is the tell: data collection ran to tens of millions of dollars while compute, headcount and research came to a few million. That inverts the usual pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → ratio and looks more like a reinforcement-learning rollout budget.
Terms in this piece · Glossary
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
It crisply explains why a model's test loss can flatline while training loss keeps dropping — an information bottleneck in the data rather than a compute/parameter ceiling — using a real drug-discovery case study, which sharpens scaling-law intuitions for anyone deciding whether to scale data vs. model.
Key quotes
“If test loss flatlines after 1.5B parameters while training loss continues to drop as you scale, that tells you that your model is limited by the amount of information in your data.”
“The budget looks like a RL rollout budget, rather than a data rich pre-training one.”
“Models trained on CELLxGENE describe the relationship between cell types and cell states, but they are not good at predicting what will happen if we make changes to RNA expression.”
