LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs

Pei-Fu Guo, Yun-Da Tsai, Chun-Chia Hsu, Kai-Xin Chen, Ya An Tsai, Kai-Wei Chang, Nanyun Peng, Mi-Yen Yeh, and Shou-De Lin, in ACL, 2026.

Download the full text

Abstract

Evaluating cross-lingual knowledge transfer in large language models is challenging, as correct answers in a target language may arise either from genuine transfer or from prior exposure during pre-training. We present LiveCLKTBench, an automated generation pipeline specifically designed to isolate and measure cross-lingual knowledge transfer. Our pipeline identifies self-contained, time-sensitive knowledge entities from real-world domains, filters them based on temporal occurrence, and verifies them against the model’s knowledge. The documents of these valid entities are then used to generate factual questions, which are translated into multiple languages to evaluate transferability across linguistic boundaries. Using LiveCLKTBench, we evaluate several LLMs across five languages and observe that cross-lingual transfer is strongly influenced by linguistic distance and often asymmetric across language directions. While larger models improve transfer, the gains diminish with scale and vary across domains. These findings provide new insights into multilingual transfer and demonstrate the value of LiveCLKTBench as a reliable benchmark for future research.

Bib Entry

@inproceedings{guo2026liveclktbench,
  title = {LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs},
  author = {Guo, Pei-Fu and Tsai, Yun-Da and Hsu, Chun-Chia and Chen, Kai-Xin and Tsai, Ya An and Chang, Kai-Wei and Peng, Nanyun and Yeh, Mi-Yen and Lin, Shou-De},
  booktitle = {ACL},
  year = {2026}
}

Related Publications

Contextual Label Projection for Cross-Lingual Structured Prediction, NAACL, 2024
Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction, ACL, 2022
Evaluating the Values of Sources in Transfer Learning, NAACL, 2021
Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training, EMNLP, 2021
Syntax-augmented Multilingual BERT for Cross-lingual Transfer, ACL, 2021
GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction, AAAI, 2021
Cross-Lingual Dependency Parsing by POS-Guided Word Reordering, EMNLP-Finding, 2020
Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages, CoNLL, 2019
Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing, EMNLP, 2019
On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing, NAACL, 2019