This repository provides Facebook Research’s original PyTorch implementation of cross-lingual language model pretraining. It includes code for monolingual and cross-lingual BERT pretraining, the XLM model for cross-lingual understanding, and applications to supervised/unsupervised machine translation and cross-lingual text classification (XNLI). It supports multi-GPU and multi-node training with various pretraining objectives including causal language modeling, masked language modeling, and translation language modeling.