The Unromantic Grind of Building a 40GB+ Clean Text Corpus

What it does

lazynlp is a Python toolkit for turning millions of raw URLs into massive, clean, monolingual text datasets. It fetches webpages, strips HTML and garbage, collapses whitespace, and uses Bloom filters to deduplicate both URLs and page content. The author claims that combining its Reddit and Gutenberg helpers can yield a corpus larger than the 40 GB WebText dataset OpenAI used for GPT-2.

The interesting bit

The library embraces brute-force pragmatism: the author admits they never bothered to parallelize the code, instead opting to run 40 separate scripts side-by-side because it was “easier.” That blunt honesty permeates the design—everything is configurable but unpretentious, from NSFW domain blacklists to n-gram overlap thresholds.

Key highlights

• Ships with helpers to extract URLs from Reddit dumps, US/Australian Gutenberg catalogs, and Wikipedia.
• Deduplicates at two stages: first URLs across files, then webpage content using configurable Bloom filters (filter_files).
• Automatically skips scraper-unfriendly or NSFW domains and extensions via built-in exclusion lists.
• Cleaning pipeline handles HTML unescaping, UTF-8 decoding, foreign character transliteration, and unprintable character removal.
• Tracks failures granularly—bad URLs, connection timeouts, empty pages, and encoding errors are logged to separate files.

Caveats

• The author explicitly notes the code is not internally parallelized; scaling means running multiple manual instances.
• Roughly 40–60% of Reddit URLs in the provided dumps are dead or scraper-unfriendly, so expect significant attrition.
• Fetching the full US Gutenberg URL list with the built-in helper can take about half a day.

Verdict

Anyone training language models from scratch and needing a DIY alternative to proprietary datasets should look here. Those looking for a fully managed, enterprise-grade crawling framework will find this too bare-bones.