Share this page:

Relation-Guided Pre-Training for Open-Domain Question Answering

Ziniu Hu, Yizhou Sun, and Kai-Wei Chang, in EMNLP-Finding, 2021.

Download the full text

Abstract

Answering complex open-domain questions requires understanding the latent relations between involving entities. However, we found that the existing QA datasets are extremely imbalanced in some types of relations, which hurts the generalization performance over questions with long-tail relations. To remedy this problem, in this paper, we propose a Relation-Guided Pre-Training (RGPT-QA) framework. We first generate a relational QA dataset covering a wide range of relations from both the Wikidata triplets and Wikipedia hyperlinks. We then pre-train a QA model to infer the latent relations from the question, and then conduct extractive QA to get the target answer entity. We demonstrate that by pretraining with propoed RGPT-QA techique, the popular open-domain QA model, Dense Passage Retriever (DPR), achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions. Particularly, we show that RGPT-QA improves significantly on questions with long-tail relations

Bib Entry

@inproceedings{hu2021relation,
  title = {Relation-Guided Pre-Training for Open-Domain Question Answering},
  author = {Hu, Ziniu and Sun, Yizhou and Chang, Kai-Wei},
  presentation_id = {https://underline.io/events/192/sessions/7932/lecture/38507-relation-guided-pre-training-for-open-domain-question-answering},
  booktitle = {EMNLP-Finding},
  year = {2021}
}

Related Publications

  1. An Integer Linear Programming Framework for Mining Constraints from Data, ICML, 2021
  2. Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs, EACL, 2021
  3. Clinical Temporal Relation Extraction with Probabilistic Soft Logic Regularization and Global Inference, AAAI, 2021
  4. PolicyQA: A Reading Comprehension Dataset for Privacy Policies, EMNLP-Finding (short), 2020
  5. GPT-GNN: Generative Pre-Training of Graph Neural Networks, KDD, 2020
  6. SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics, ACL, 2020
  7. Building Language Models for Text with Named Entities, ACL, 2018
  8. Learning from Explicit and Implicit Supervision Jointly For Algebra Word Problems, EMNLP, 2016