The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks

Nikil Roashan Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang, in ACL (short), 2023.

Abstract

How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given language model? In this work, we study this question by contrasting social biases with non-social biases stemming from choices made during dataset construction that might not even be discernible to the human eye. To do so, we empirically simulate various alternative constructions for a given benchmark based on innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias. On two well-known social bias benchmarks (Winogender and BiasNLI) we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models. We hope these troubling observations motivate more robust measures of social biases.

Bib Entry

@inproceedings{roashan2023tail,
  author = {Selvam, Nikil Roashan and Dev, Sunipa and Khashabi, Daniel and Khot, Tushar and Chang, Kai-Wei},
  title = {The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks},
  presentation_id = {https://underline.io/events/395/posters/15337/poster/76963-the-tail-wagging-the-dog-dataset-construction-biases-of-social-bias-benchmarks},
  booktitle = {ACL (short)},
  year = {2023}
}

Related Publications

A Meta-Evaluation of Measuring LLM Misgendering, COLM 2025, 2025
White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs, ACL, 2025
Controllable Generation via Locally Constrained Resampling, ICLR, 2025
On Localizing and Deleting Toxic Memories in Large Language Models, NAACL-Finding, 2025
Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification, EMNLP-Finding, 2024
Mitigating Bias for Question Answering Models by Tracking Bias Influence, NAACL, 2024
Are you talking to ['xem'] or ['x', 'em']? On Tokenization and Addressing Misgendering in LLMs with Pronoun Tokenization Parity, NAACL-Findings, 2024
Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems, EMNLP-Finding, 2023
Kelly is a Warm Person, Joseph is a Role Model: Gender Biases in LLM-Generated Reference Letters, EMNLP-Findings, 2023
Factoring the Matrix of Domination: A Critical Review and Reimagination of Intersectionality in AI Fairness, AIES, 2023
How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?, EMNLP (Short), 2022
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations, ACL (short), 2022
Societal Biases in Language Generation: Progress and Challenges, ACL, 2021
"Nice Try, Kiddo": Investigating Ad Hominems in Dialogue Responses, NAACL, 2021
BOLD: Dataset and metrics for measuring biases in open-ended language generation, FAccT, 2021
Towards Controllable Biases in Language Generation, EMNLP-Finding, 2020
The Woman Worked as a Babysitter: On Biases in Language Generation, EMNLP (short), 2019

The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks