Vietnamese NLP gets its own large-scale RoBERTa

What it does

PhoBERT offers pre-trained base and large language models built exclusively for Vietnamese using the RoBERTa training procedure. It provides contextualized embeddings and fine-tuning baselines for standard Vietnamese NLP tasks—named-entity recognition, part-of-speech tagging, dependency parsing, and natural language inference—accessible through Hugging Face transformers or fairseq.

The interesting bit

The model was trained monolingually on up to 140GB of Vietnamese text, but it comes with a rigid preprocessing contract: input must be word-segmented with underscores between syllables (e.g., nghiên_cứu_viên), ideally using the same RDRSegmenter pipeline that prepared its training corpus. That tight coupling between a specific Vietnamese word segmenter and the tokenizer is unusual in the BERT family.

Key highlights

• First public large-scale monolingual language models pre-trained for Vietnamese, with base, large, and a newer base-v2 variant trained on 140GB of text.
• Outperforms previous monolingual and multilingual approaches on four downstream tasks according to the EMNLP 2020 Findings paper.
• Accessible via standard transformers AutoModel/AutoTokenizer APIs and fairseq.
• Original base and large models use an MIT license; the updated base-v2 uses GNU Affero GPL v3.

Caveats

• Input must be pre-word-segmented; raw text will be tokenized incorrectly without passing through a segmenter like RDRSegmenter first.
• Maximum sequence length is hard-capped at 256 tokens.
• Fast tokenizer support is still pending in a side branch and has not landed in mainline transformers.

Verdict

A solid starting point if you are building Vietnamese-specific NLP pipelines. Look elsewhere if you need multilingual coverage or contexts longer than 256 tokens.