Intel / article

Capability-Driven Multimodal Scaling Law

Source: arxiv.org
Author: Ziran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai
Date: Published Aug 4, 2026

Context

Text-only test results may help researchers choose a before training it to understand images. The Capability-Driven Scaling Law compresses those test results into a capability score, then models how that capability transfers to image-and-text tasks and responds to additional training data. The authors trained more than 150 vision-language models using 34 language-model foundations under a controlled recipe. They report successful predictions for held-out model families and extrapolation from foundations of up to 8 billion parameters to 72-billion-parameter models. The proposed use is to narrow expensive training comparisons using measurable starting capabilities. Research abstract.

Key takeaways · AI-distilled

A low-dimensional capability score extracted from an LLM's textual benchmarks via PCA predicts its downstream VLM accuracy across model families — modeled with a per-backbone transfer rate plus an absorption rate for data-scaling efficiency.
The law was fit on 150+ VLMs trained over 34 LLMs from 7 families under one controlled recipe, extrapolates from sub-8B models to 72B backbones, and holds on entirely held-out families.
Base LLMs make better VLM backbones than their instruction-tuned counterparts — higher absorption rates and lower data-scaling decay.
Certain textual benchmarks correlate NEGATIVELY with multimodal performance, exposing latent -gaming in backbone models.
Practical effect: backbone selection becomes a quantitative prediction instead of an expensive trial-and-error training sweep.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

Key quotes

“We propose the Capability-Driven Multimodal Scaling Law, the first cross-family framework that predicts VLM benchmark accuracy from directly observable textual capability.”

“The framework turns backbone selection from costly empirical sweeps into a principled, quantitative decision.”

Read the source arxiv.org