Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning

Yixin Wan, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Rahul Gupta, in EMNLP-Finding, 2025.

Download the full text

Abstract

Large Language Model (LLM) unlearning has recently gained significant attention because of the need to remove unwanted information such as private or copyrighted content. Conventional unlearning approaches indiscriminately update model parameters to forget all tokens, including common tokens that carry general knowledge. This paper highlights that not every token needs forgetting and proposes Selective Unlearning (SU), which identifies a critical subset of tokens within the forgetting set that is relevant to unwanted information and unlearns only those tokens. Experiments on two benchmarks and six baseline unlearning algorithms show that SU achieves effective unlearning on the targeted forget data while significantly preserving the model’s utility in the retaining set.

Bib Entry

@inproceedings{wan2025not,
  title = {Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning},
  author = {Wan, Yixin and Ramakrishna, Anil and Chang, Kai-Wei and Cevher, Volkan and Gupta, Rahul},
  booktitle = {EMNLP-Finding},
  year = {2025}
}