Generalized Visual Language Models

A language model can gain access to images while its existing parameters remain unchanged. Lilian Weng's 2022 survey explains this through ClipCap, an image-captioning system. An image encoder converts a picture into numerical features, and a trained mapping network translates those features into a sequence the language model can use before generating a caption.

In the frozen configuration described, training changes that connecting network while leaving both the image encoder and language model fixed. The new learning happens at the interface between them. This offers a concrete way to reuse existing visual and language capabilities for captioning without retraining both underlying models.

“One straightforward approach to fuse visual information into language models is to treat images as normal text tokens and train the model on a sequence of joint representations of both text and images. Precisely, images are divided into multiple smaller patches and each patch is treated as one “token” in the input sequence.”

Lilian Weng

“According to ablation experiments, the most important configuration is to fuse visual information early on into the transformer layers and to pretrain the model on the COCO caption dataset. Initialization from a pre-trained BERT and the adoption of the sentence-image prediction training objective have relatively small impacts.”

Lilian Weng

“According to ablation studies, it is important to have both image-text and text-only data for training. The PrefixLM objective outperforms both span corruption and naive LM.”

Lilian Weng

“Experiments showed that fine-tuning the pre-trained LM interestingly leads to worse performance on VQA tasks.”

Lilian Weng

“Even though ClipCap only trains such a minimum set of parameters, it still achieves decent performance on image captioning tasks, comparable with SoTA at the time (e.g. Oscar , VLP , BUTD ).”

Lilian Weng