A janitorial service for Chinese chatbot training data

What it does This project is a data aggregator and sanitizer for Chinese conversational AI. It collects eight publicly available Chinese chat corpora—from Chatterbot and Douban to Weibo and PTT—then extracts them, converts Traditional Chinese to Simplified, and normalizes everything into uniform single-turn query \t answer TSV files. The goal is to spare you from chasing scattered download links and wrestling with idiosyncratic formats.

The interesting bit The real work here is janitorial, not algorithmic. The author flattens multi-turn dialogues into isolated pairs, handles mixed encodings and traditional characters, and is refreshingly blunt about the dirt left on the floor: noise, forum slang, and even indecent exchanges are labeled as-is rather than glossed over.

Key highlights

• Aggregates 8 sources including Douban (3.52M pairs), Weibo (4.43M), and TV subtitles (2.74M)
• Normalizes multi-turn threads into single-turn TSV rows with Simplified Chinese
• Provides a single Python pipeline to process all corpora at once
• Source provenance is documented, though some links rely on Baidu Pan or direct IPs
• README is honest about quality: outputs are “roughly usable,” not pristine

Caveats

• Several corpora carry significant noise, slang, and offensive content (the Xiaohuangji set includes indecent dialogue)
• Some source distribution links look fragile (Baidu Pan shares and raw IPs)
• Multi-turn conversations are flattened into isolated pairs, so context is lost

Verdict Worth bookmarking if you are prototyping a Chinese chatbot and need volume fast. Skip it if you require clean, production-grade dialogue data without heavy additional filtering.