Indic NLP Without Training Your Own Embeddings

What it does iNLTK is a Python library that wraps pre-trained ULMFiT and TransformerXL language models for over a dozen Indic languages (plus English). It exposes standard NLP tasks—tokenization, embeddings, similarity, and classification—through a unified API so you don’t have to train models from scratch on Wikipedia articles. The library also handles code-mixed text like Hinglish and Tanglish.

The interesting bit Rather than inventing new architectures, the author trained separate fastai models for each language and bundled them behind a single interface. The unusual angle is the inclusion of Roman-script code-mixed dialects alongside native scripts, which most mainstream NLP toolkits treat as an afterthought.

Key highlights

• Supports 14 native languages including Hindi, Bengali, Tamil, and Sanskrit, plus English
• Covers code-mixed variants: Hinglish, Tanglish, and Manglish in Latin script
• Bundles pre-trained ULMFiT and TransformerXL models with reported perplexity and classification benchmarks
• Paper accepted at EMNLP-2020’s NLP-OSS workshop
• Underlying models live in separate per-language repositories

Caveats

• The README is mostly a benchmark spreadsheet; the actual API surface is vague unless you read the external ReadTheDocs documentation
• The project is largely glue code around fastai/ULMFiT and TransformerXL

Verdict Worth a look if you need quick embeddings or classification for Indic text and don’t want to wrangle fastai yourself. Skip it if you already maintain your own transformer pipeline or only need basic tokenization.