grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
Why it matters
A structured survey of how to graft visual understanding onto pre-trained language models, covering the design choices behind modern VLMs — useful groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → for anyone building or fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition →multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition → AI systems.
Key quotes
“One straightforward approach to fuse visual information into language models is to treat images as normal text tokens and train the model on a sequence of joint representations of both text and images. Precisely, images are divided into multiple smaller patches and each patch is treated as one “token” in the input sequence.”
Lilian Weng
“According to ablation experiments, the most important configuration is to fuse visual information early on into the transformer layers and to pretrain the model on the COCO caption dataset. Initialization from a pre-trained BERT and the adoption of the sentence-image prediction training objective have relatively small impacts.”
Lilian Weng
“According to ablation studies, it is important to have both image-text and text-only data for training. The PrefixLM objective outperforms both span corruption and naive LM.”
Lilian Weng
“Experiments showed that fine-tuning the pre-trained LM interestingly leads to worse performance on VQA tasks.”
Lilian Weng
“Even though ClipCap only trains such a minimum set of parameters, it still achieves decent performance on image captioning tasks, comparable with SoTA at the time (e.g. Oscar , VLP , BUTD ).”
