Control Large Language Models via Divide and Conquer

Bingxuan Li, Yiwei Wang, Tao Meng, Kai-Wei Chang, and Nanyun Peng, in EMNLP, 2024.

Download the full text

Abstract

Bib Entry

@inproceedings{li2024llms,
  title = {Control Large Language Models via Divide and Conquer},
  author = {Li, Bingxuan and Wang, Yiwei and Meng, Tao and Chang, Kai-Wei and Peng, Nanyun},
  booktitle = {EMNLP},
  abstrct = {This paper investigates the capability of LLMs on controllable generation with prompt-based controlling, focusing on Lexically Constrained Generation (LCG). We systematically evaluate the performance of LLMs on satisfying lexical constraints with prompt-based controlling, as well as their efficacy in downstream applications. We identified three key reasons that highlight the limitations of LLMs in LCG, including (1) position bias, where LLMs tend to satisfy constraints that appear in specific positions within the input; (2) low responsiveness to control decoding parameters, which minimally impact the performance of LLMs; and (3) struggle with handling the inherent complexity of certain constraints (e.g. compound word). We conclude that black-box LLMs face significant challenges in consistently satisfying lexical constraints with prompt-based controlling. To address this bottleneck, we introduce the Divide and Conquer Generation strategy, effective for both white-box and black-box LLMs, to enhance LLMs performance in LCG tasks, which demonstrates over 90% improvement on success rate in the most challenging LCG task. Our analysis aims to provide valuable insights into the performance of LLMs in LCG with prompt-based controlling, and our proposed strategy offers a pathway to more sophisticated and customized text generation applications.},
  year = {2024}
}

Related Publications

Re-ReST: Reflection-Reinforced Self-Training for Language Agents, EMNLP, 2024
Agent Lumos: Unified and Modular Training for Open-Source Language Agents, ACL, 2024
Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension, ICML, 2024
TrustLLM: Trustworthiness in Large Language Models, ICML, 2024
The steerability of large language models toward data-driven personas, NAACL, 2024
AI-Assisted Summarization of Radiologic Reports: Evaluating GPT3davinci, BARTcnn, LongT5booksum, LEDbooksum, LEDlegal, and LEDclinical, American Journal of Neuroradiology, 2024
Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis, EACL-Findings, 2024
Few-Shot Representation Learning for Out-Of-Vocabulary Words, ACL, 2019
Learning Word Embeddings for Low-resource Languages by PU Learning, NAACL, 2018
Co-training Embeddings of Knowledge Graphs and Entity Descriptions for Cross-lingual Entity Alignment, IJCAI, 2018
Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context, ACL RepL4NLP Workshop, 2017