Pre-trained Arabic Transformers that handle dialects and emojis

What it does

AraBERT is a family of pre-trained Transformer models—BERT, GPT-2, and ELECTRA variants—trained from scratch on a 77GB Arabic corpus spanning Wikipedia, news, and web text. The project publishes checkpoints in multiple sizes and frameworks, plus a preprocessing toolkit that handles segmentation, diacritics, and dialectal text cleanup before inference or fine-tuning. It also includes Twitter-specific models with emoji-aware vocabularies.

The interesting bit

The real craft is in the preprocessing layer: it can reverse its own changes with an unpreprocess() function, which is especially useful for making AraGPT2 output look natural again. The team also fixed a subtle vocabulary bug in the original AraBERT where punctuation and numbers were fused to words, and they trained dedicated models on roughly 60 million Arabic tweets to handle dialect and emoji-heavy text.

Key highlights

• Six AraBERT variants (base/large, v0.2/v2, Twitter-tuned) with up to 371M parameters.
• AraGPT2 scales up to a 1.46B-parameter “mega” model, plus a detector variant.
• AraELECTRA ships separate generator (60M) and discriminator (135M) checkpoints.
• All models are on HuggingFace in PyTorch, TensorFlow 1, and TensorFlow 2 formats.
• The ArabertPreprocessor handles Farasa segmentation, tashkeel stripping, and emoji retention depending on the target model.

Caveats

• AraBERT v1 and v2 expect Farasa segmentation, while v0.2 does not; the README explicitly warns that you must match the preprocessor to the model name.
• The Twitter models were trained with a max sequence length of 64 for only one epoch, which may limit their depth of context.
• The default preprocessor automatically replaces slashes with dashes and maps Hindi digits to Arabic digits for v0.2, AraELECTRA, and AraGPT2, which may not suit all downstream tasks.

Verdict

Worth a look if you are building Arabic NLP pipelines and need models that actually understand right-to-left morphology, dialects, and diacritics. Skip it if your text is strictly English or if you are looking for a general multilingual model rather than an Arabic-native stack.