Chinese text mining that learns your domain’s nicknames

What it does

HarvestText is a Python toolkit for Chinese text mining and preprocessing that leans almost entirely on unsupervised or weakly supervised methods. It handles the usual suspects—segmentation, named entity recognition, dependency parsing, sentiment analysis, keyword extraction, and summarization—but wraps them around a core idea: you can inject domain knowledge (entity aliases, custom types, field-specific dictionaries) to make the pipeline aware of your corpus. It also scrubs real-world noise like Weibo @-mentions, emoji, URLs, HTML escapes, and traditional Chinese characters.

The interesting bit

The standout feature is entity linking with alias discovery. You can feed it a dictionary mapping standard names to nicknames—e.g., mapping “武球王” back to “武磊”—and the segmenter will preserve them as typed tokens rather than shredding them into meaningless pieces. If you don’t have a dictionary, it can even suggest aliases automatically from raw text, which makes it unusually handy for social media, sports commentary, or fiction where standard tokenizers typically fall apart.

Key highlights

• Unsupervised sentiment analysis that bootstraps from a handful of seed words rather than a labeled dataset.
• Built-in domain dictionaries for IT, finance, food, and law, plus generic stop-word and sentiment lists.
• Text cleaning handles Weibo formatting, URL/email stripping, HTML entity normalization, and traditional-to-simplified conversion.
• New-word discovery using statistical rules to catch vocabulary that standard segmenters miss.
• Fact extraction and a toy QA pipeline built from dependency-parsing triples (explicitly billed as a demo).

Caveats

• Several features, including named entity recognition and dependency parsing, rely on pyhanlp and require Python ≤ 3.8; the README warns these extra dependencies may fail to install and must be added manually.
• The included QA system and knowledge-graph demo are explicitly noted as “效果有待提升，仅作为示例” (effectiveness needs improvement, for demonstration only).
• English support is minimal; the library is designed primarily for Chinese text.

Verdict

Grab this if you need to quickly explore or preprocess messy Chinese corpora—especially social media, novels, or vertical-domain documents—without curating training data. Look elsewhere if you need production-grade English NLP or state-of-the-art deep-learning models.