BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Junnan Li
and
Dongxu Li
and
Silvio Savarese
and
Steven Hoi
arXiv e-Print archive - 2023 via Local arXiv
Keywords:
cs.CV
First published: 2024/12/13 (just now) Abstract: The cost of vision-and-language pre-training has become increasingly
prohibitive due to end-to-end training of large-scale models. This paper
proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps
vision-language pre-training from off-the-shelf frozen pre-trained image
encoders and frozen large language models. BLIP-2 bridges the modality gap with
a lightweight Querying Transformer, which is pre-trained in two stages. The
first stage bootstraps vision-language representation learning from a frozen
image encoder. The second stage bootstraps vision-to-language generative
learning from a frozen language model. BLIP-2 achieves state-of-the-art
performance on various vision-language tasks, despite having significantly
fewer trainable parameters than existing methods. For example, our model
outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable
parameters. We also demonstrate the model's emerging capabilities of zero-shot
image-to-text generation that can follow natural language instructions.
This paper has a way to leverage pre-trained Vision Language encoders to do VL tasks such as VQA, and Image Captioning.
To have a good VL model, the modality gap must be reduced. In this paper, they proposed a Q-Former which is a Transformer module that is trained first with a frozen image encoder, then trained with this frozen image encoder and a frozen text encoder (from a Large Language Model).
https://i.imgur.com/rQ3V3oQ.png
The reason why the Q-Former needs to train in two stages is:
(1) Trained with frozen image encoder to learn the most informative visual features.
https://i.imgur.com/gshAy1p.png
(2) Trained with frozen text encoder to learn the visual features related to the textual feature the most.
https://i.imgur.com/gPz40GC.png