Multilingual Transformer
Multilingual Transformer (Cross-lingual Pre-trained Language Model) · Also known as: multilingual LM, cross-lingual transformer, mBERT-style model, multilingual pre-trained model
A multilingual transformer is a pre-trained language model built on the transformer architecture and trained jointly on text from dozens to over one hundred languages. Models such as mBERT and XLM-RoBERTa learn shared cross-lingual representations, enabling zero-shot or few-shot transfer: a model fine-tuned on English data can often be applied directly to French, German, Arabic, or Chinese without language-specific labels.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
+10 more
When to use it
Use a multilingual transformer when your data spans multiple languages, when labelled data is scarce in some languages but available in others, or when you need a single unified model to handle multilingual input at inference time. It is especially valuable for zero-shot or few-shot cross-lingual transfer in NLP tasks such as classification, NER, and question answering. Avoid it when you are working in a single high-resource language with abundant labelled data — a monolingual model trained on that language alone will typically outperform a multilingual one. Also avoid it when compute or memory budgets are very tight, as large multilingual models are resource-intensive.
Strengths & limitations
- Enables zero-shot and few-shot transfer: a model fine-tuned on one language generalises to others without additional labelled data.
- A single model handles many languages, simplifying deployment and maintenance compared to separate language-specific models.
- Strong baselines for low-resource languages that benefit from shared representations with higher-resource relatives.
- Well-supported by open-source ecosystems (Hugging Face Transformers) with pretrained checkpoints for mBERT, XLM-R, mT5, and others.
- Contextual embeddings capture nuanced meaning that static word embeddings cannot represent.
- The multilingual curse: performance on each individual language tends to be lower than a comparable monolingual model trained solely on that language.
- Large model sizes (hundreds of millions to billions of parameters) require substantial GPU memory and compute for both fine-tuning and inference.
- Coverage is uneven — low-resource languages with little training data in the pre-training corpus receive weaker representations.
- Tokenisation of morphologically complex or script-diverse languages may be suboptimal with a shared vocabulary.
Frequently asked
Is a multilingual transformer the same as a machine translation model?
No. A multilingual transformer learns shared representations across languages but does not translate text. It produces language-specific outputs (e.g., a classification label) that are informed by cross-lingual understanding, whereas a translation model generates text in a target language.
Which pretrained checkpoint should I start with?
XLM-RoBERTa-large is a strong general-purpose starting point for most NLP tasks. For tasks requiring generation or translation, mT5 is a good alternative. For very constrained compute budgets, XLM-RoBERTa-base offers a reasonable trade-off.
How much fine-tuning data do I need?
Even a few hundred labelled examples in a source language can produce useful zero-shot transfer to related languages. Adding even a small number of target-language examples during fine-tuning typically yields significant improvements.
What is the multilingual curse and how can I mitigate it?
The multilingual curse refers to the observation that sharing capacity across many languages reduces per-language performance compared to monolingual models. Mitigation strategies include using a larger model, fine-tuning on target-language data when available, or using language-adaptive fine-tuning (LAFT).
Can I use a multilingual transformer for non-European languages?
Yes — models like XLM-RoBERTa cover Arabic, Chinese, Japanese, Korean, Hindi, Swahili, and many others. However, coverage and quality vary: inspect the pre-training language distribution for your specific language before relying on the model.
Sources
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, pp. 4171–4186. Association for Computational Linguistics. DOI: 10.18653/v1/N19-1423 ↗
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of ACL 2020, pp. 8440–8451. Association for Computational Linguistics. DOI: 10.18653/v1/2020.acl-main.747 ↗
How to cite this page
ScholarGate. (2026, June 3). Multilingual Transformer (Cross-lingual Pre-trained Language Model). ScholarGate. https://scholargate.app/en/deep-learning/multilingual-transformer
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- BERT-based ClassificationDeep learning↔ compare
- Multilingual Sentence EmbeddingsDeep learning↔ compare
- RoBERTa-based ClassificationDeep learning↔ compare
- Sentence EmbeddingsDeep learning↔ compare