A 4B-parameter foundation model built from scratch for Vietnamese

What it does

PhoGPT is a family of Vietnamese generative models—a 3.7B-parameter base model and an instruction-tuned chat variant—trained from scratch on a 102B-token Vietnamese corpus. The base model handles 8192-token contexts with a compact 20K vocabulary, while the chat version adds fine-tuning on 70K instruction pairs and 290K conversations. It is designed to run on modest hardware, loading in float16 with roughly 7GB of GPU memory, and supports popular inference engines like vLLM and llama.cpp.

The interesting bit

Instead of fine-tuning a multilingual model, PhoGPT is trained from scratch on Vietnamese text alone—a 102B-token monolingual corpus—rather than being adapted from English-centric weights. The authors claim it outperforms prior open-source Vietnamese models, though the README leaves the detailed benchmark numbers to the linked technical report and a Google Sheet.

Key highlights

• 3.7B parameters, 8192 context length, and a 20K vocab—small enough to run on a single consumer GPU in float16 (≈7GB VRAM).
• Two variants: a base causal LM (PhoGPT-4B) and an instruction-following chat model (PhoGPT-4B-Chat).
• Pre-trained from scratch on 482GB of Vietnamese text for 2 epochs; the chat variant adds 360K instruction and conversation examples.
• Supports vLLM, Text Generation Inference, llama.cpp (with published GGUF weights), and Hugging Face transformers.
• Fine-tuning recipes provided via an llm-foundry YAML config and a sample instruction-following dataset.

Caveats

• The model struggles with reasoning, coding, and mathematics, and can produce biased, harmful, or factually incorrect outputs.
• The README touts “superior performance” but does not include quantitative results inline; you have to dig into the arXiv paper or a Google Sheet for evidence.
• trust_remote_code=True is required to load the model in transformers, which means you are executing unaudited custom model code.

Verdict

Worth a look if you are building Vietnamese-language chatbots or NLP pipelines and need a compact, natively trained model. Skip it if your use case demands strong reasoning, code generation, or rigorous mathematical accuracy.