Share this page:

Syntax-augmented Multilingual BERT for Cross-lingual Transfer

Wasi Ahmad, Haoran Li, Kai-Wei Chang, and Yashar Mehdad, in ACL, 2021.

Code

Download the full text

Abstract

In recent years, we have seen a colossal effort in pre-training multilingual text encoders using large-scale corpora in many languages to facilitate cross-lingual transfer learning. However, due to typological differences across languages, the cross-lingual transfer is challenging. Nevertheless, language syntax, e.g., syntactic dependencies, can bridge the typological gap. Previous works have shown that pretrained multilingual encoders, such as mBERT (Devlin et al., 2019), capture language syntax, helping cross-lingual transfer. This work shows that explicitly providing language syntax and training mBERT using an auxiliary objective to encode the universal dependency tree structure helps cross-lingual transfer. We perform rigorous experiments on four NLP tasks, including text classification, question answering, named entity recognition, and taskoriented semantic parsing. The experiment results show that syntax-augmented mBERT improves cross-lingual transfer on popular benchmarks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across all languages. In the generalized transfer setting, the performance boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA.

Bib Entry

@inproceedings{ahmad2021syntax,
  title = {Syntax-augmented Multilingual BERT for Cross-lingual Transfer},
  author = {Ahmad, Wasi and Li, Haoran and Chang, Kai-Wei and Mehdad, Yashar},
  booktitle = {ACL},
  year = {2021}
}

Related Publications

  1. Contextual Label Projection for Cross-Lingual Structured Prediction, NAACL, 2024
  2. Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction, ACL, 2022
  3. Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training, EMNLP, 2021
  4. Evaluating the Values of Sources in Transfer Learning, NAACL, 2021
  5. GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction, AAAI, 2021
  6. Cross-Lingual Dependency Parsing by POS-Guided Word Reordering, EMNLP-Finding, 2020
  7. Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages, CoNLL, 2019
  8. Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing, EMNLP, 2019
  9. On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing, NAACL, 2019