Automating the least fun part of content moderation: data collection

What it does These are shell scripts that run inside Docker to scrape tens of thousands of images from subreddits and other sources, sorting them into five loosely defined categories—porn, hentai, sexy, neutral, and drawings—to build labeled training and test sets for image classifiers. The pipeline wraps the ripme application for URL harvesting, optionally pulls SFW images from Danbooru2018 and Caltech256, and handles the tedious hygiene work like stripping corrupted JPEGs and carving out a holdout set.

The interesting bit Most ML repositories treat data collection as a footnote; this one treats it as the entire product. The author already ran the initial URL scrape, so you inherit a head start rather than waiting for ripme to trawl Reddit overnight, and the README is admirably upfront about the dataset being noisy and the categories fuzzy.

Key highlights

• Five-way taxonomy that tries to separate explicit content from suggestive content, and real photography from illustrations
• Pre-harvested URL lists in raw_data so you can skip the multi-hour scraping step
• Optional augmentation with curated SFW sources (Danbooru2018 for anime, Caltech256 for neutral objects)
• Built-in train/test split via 6_create_test_.sh, defaulting to 2,000 images per class
• Includes a train_model.ipynb notebook that reportedly reaches 91% accuracy with fastai

Caveats

• The README explicitly warns that the dataset is noisy
• Categories are “loosely defined,” and the included confusion matrix shows predictable overlap between drawings and hentai, plus porn and sexy
• Entirely dependent on third-party scrapers and external sources that can change or disappear

Verdict Worth a look if you need a quick, dirty bootstrap for content-moderation classifiers and don’t mind cleaning up after shell scripts. Skip it if you require rigorously labeled, rights-cleared data with precise taxonomies.