• A low-dimensional capability score extracted from an LLM's textual benchmarks via PCA predicts its downstream VLM accuracy across model families — modeled with a per-backbone transfer rate plus an absorption rate for data-scaling efficiency.
• The law was fit on 150+ VLMs trained over 34 LLMs from 7 families under one controlled recipe, extrapolates from sub-8B models to 72B backbones, and holds on entirely held-out families.
• Base LLMs make better VLM backbones than their instruction-tuned counterparts — higher absorption rates and lower data-scaling decay.
• Certain textual benchmarks correlate NEGATIVELY with multimodal performance, exposing latent benchmark-gaming in backbone models.
• Practical effect: backbone selection becomes a quantitative prediction instead of an expensive trial-and-error training sweep.

• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
• benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

The paper shows a low-dimensional textual capability score (via PCA) predicts multimodal performance across model families and scales up to 72B, letting practitioners pick an LLM backbone for a VLM via prediction instead of expensive trial-and-error training runs.

• articleMaking LLMs faster without sacrificing accuracywww.amazon.science
• articleScaling Laws for Neural Language ModelsJared Kaplan et al.
• articleMultimodality and Large Multimodal Models (LMMs)Chip Huyen

Checking sign-in…

Loading comments…