Teaching multimodal models to zoom internally, without the tool call
- Source
- x.com
- Date
🔍MLLMs are nearsighted: zooming helps but kills speed. What if we teach them to "see" fine details without zooming? Introducing Region-to-Image Distillation(R2I): we train models to internalize "zooming". 🏆 ZwZ-8B achieves SOTA performance on fine-grained perception with zero tool calls. 📊 Plus ZoomBench: human-AI collaboration construction, dual-view "zooming gap" metric Dive into our joint efforts: 📑Paper: https://t.co/tdEn2QwoKD 🔗Code: https://t.co/b8S4HqcdPT ⚙️ Model&Data: https://t.co/5ZDak0L3C8 #OpenSource #MLLM #LLMs #inclusionAI
Context
models are commonly described as nearsighted: they can zoom into a region of an image to read fine detail, but that extra tool call slows the whole pipeline down. Ant Open Source, under InclusionAI, introduces Region-to-Image Distillation (R2I), a training method meant to have the model internalize what zooming would have shown it, so it can read fine detail without making the extra call. The resulting model, ZwZ-8B, is reported to reach state-of-the-art fine-grained perception performance with zero tool calls, and the team also releases ZoomBench, a built with human-AI collaboration to measure the gap between what a model sees at normal resolution and what zooming would reveal.
- multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Checking sign-in…
Loading comments…