Back to news
arXiv cs.CL · 2026-10-22 00:00 UTC
research

How to Use Parallel Data for Multilingual Pretraining

arXiv:2610.28838v1 Announce Type: new Abstract: High-quality human-translated parallel corpora can improve cross-lingual transfer, but are typically too small for large-scale pretraining. Machine-translated web corpora help overcome this limitation, making parallel data available at pretraining scale. Standard multilingual pretraining, however, samples source documents and their translations independently, without explicitly using their parallel structure. We investigate how to use the parallel structure of the data to improve multilingual pretraining by concatenating each document with its tr

Why it matters

Using parallel data in pretraining can significantly improve multilingual performance and cross-lingual transfer, lowering the cost to support new languages.

Read the original story

Published to Cognify News · Week 43, 2026