Share this page:

VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval

Di Wu, Yixin Wan, and Kai-Wei Chang, in ACL, 2026.

Code

Download the full text

Abstract

Text-to-image retrieval (T2I retrieval) remains challenging because cross-modal embeddings often behave as bags of concepts, underrepresenting structured visual relationships such as pose and viewpoint. We proposeVisualize-then-Retrieve (VisRet), a retrieval paradigm that mitigates this limitation of cross-modal similarity alignment. VisRet first projects textual queries into the image modality via T2I generation, then performs retrieval within the image modality to bypass the weaknesses of cross-modal retrievers in recognizing subtle visual-spatial features. Across four benchmarks (Visual-RAG, INQUIRE-Rerank, Microsoft COCO, and our new Visual-RAG-ME featuring multi-entity comparisons), VisRet substantially outperforms cross-modal similarity matching and baselines that recast T2I retrieval as text-to-text similarity matching, improving nDCG@30 by 0.125 on average with CLIP as the retriever and by 0.121 with E5-V. For downstream question answering, VisRet increases accuracy on Visual-RAG and Visual-RAG-ME by 3.8% and 15.7% in top-1 retrieval, and by 3.9% and 11.1% in top-10 retrieval. Ablation studies show compatibility with different T2I instruction LLMs, T2I generation models, and downstream LLMs. VisRet provides a simple yet effective perspective for advancing in text-image retrieval. Our code and the new benchmark are publicly available at https://github.com/xiaowu0162/Visualize-then-Retrieve.

Bib Entry

@inproceedings{wu2026visret,
  title = {VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval},
  author = {Wu, Di and Wan, Yixin and Chang, Kai-Wei},
  booktitle = {ACL},
  year = {2026}
}

Related Publications

  1. HoneyBee: Data Recipes for Vision-Language Reasoners, CVPR, 2026
  2. MotionEdit: Benchmarking and Learning Motion-Centric Image Editing, CVPR, 2026
  3. LaViDa: A Large Diffusion Language Model for Multimodal Understanding, NeurIPS, 2025
  4. PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding, NeurIPS, 2025
  5. SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation, ICLR, 2025
  6. Verbalized Representation Learning for Interpretable Few-Shot Generalization, ICCV, 2025
  7. STIV: Scalable Text and Image Conditioned Video Generation, ICCV, 2025
  8. Contrastive Visual Data Augmentation, ICML, 2025
  9. SYNTHIA: Novel Concept Design with Affordance Composition, ACL, 2025
  10. Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation, Nature Communications, 2025
  11. Enhancing Large Vision Language Models with Self-Training on Image Comprehension, NeurIPS, 2024
  12. CoBIT: A Contrastive Bi-directional Image-Text Generation Model, ICLR, 2024
  13. DesCo: Learning Object Recognition with Rich Language Descriptions, NeurIPS, 2023
  14. Text Encoders are Performance Bottlenecks in Contrastive Vision-Language Models, EMNLP, 2023
  15. "What's 'up' with vision-language models? Investigating their struggle to understand spatial relations.", EMNLP, 2023
  16. MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models, ACL (short), 2023
  17. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge, CVPR, 2023
  18. Grounded Language-Image Pre-training, CVPR, 2022
  19. How Much Can CLIP Benefit Vision-and-Language Tasks?, ICLR, 2022