How Much Can CLIP Benefit Vision-and-Language Tasks?

Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutz, in ICLR, 2022.

Abstract

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks.

Source Code

Bib Entry

@inproceedings{shen2022how,
  title = { How Much Can CLIP Benefit Vision-and-Language Tasks? },
  author = {Shen, Sheng and Li, Liunian Harold and Tan, Hao and Bansal, Mohit and Rohrbach, Anna and Chang, Kai-Wei and Yao, Zhewei and Keutz, Kurt},
  booktitle = {ICLR},
  year = {2022}
}

Related Publications

HoneyBee: Data Recipes for Vision-Language Reasoners, CVPR, 2026
MotionEdit: Benchmarking and Learning Motion-Centric Image Editing, CVPR, 2026
LaViDa: A Large Diffusion Language Model for Multimodal Understanding, NeurIPS, 2025
PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding, NeurIPS, 2025
STIV: Scalable Text and Image Conditioned Video Generation, ICCV, 2025
Verbalized Representation Learning for Interpretable Few-Shot Generalization, ICCV, 2025
Contrastive Visual Data Augmentation, ICML, 2025
SYNTHIA: Novel Concept Design with Affordance Composition, ACL, 2025
SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation, ICLR, 2025
Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation, Nature Communications, 2025
Enhancing Large Vision Language Models with Self-Training on Image Comprehension, NeurIPS, 2024
CoBIT: A Contrastive Bi-directional Image-Text Generation Model, ICLR, 2024
DesCo: Learning Object Recognition with Rich Language Descriptions, NeurIPS, 2023
"What's 'up' with vision-language models? Investigating their struggle to understand spatial relations.", EMNLP, 2023
Text Encoders are Performance Bottlenecks in Contrastive Vision-Language Models, EMNLP, 2023
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models, ACL (short), 2023
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge, CVPR, 2023
Grounded Language-Image Pre-training, CVPR, 2022

How Much Can CLIP Benefit Vision-and-Language Tasks?