Turn a list of URLs into a training dataset overnight

What it does

img2dataset ingests lists of image URLs (and optional captions) and emits sharded, resized archives ready for training. It handles the full fetch-and-munge pipeline—downloading, decoding, resizing, re-encoding, and metadata extraction—so you don’t have to maintain yet another bespoke scraper. The README claims it can chew through 100 million URLs in roughly 20 hours on a single machine.

The interesting bit

Most scrapers just dump files into a folder; this one treats dataset assembly as a proper ETL job. It writes directly to ML-friendly formats like WebDataset, TFRecord, or Parquet, records EXIF data and download status for every image, and scales out via PySpark or Ray. It also respects X-Robots-Tag: noai opt-out headers by default—a rare bit of politeness in the scraping world.

Key highlights

• Claims throughput of 100M URLs in 20 hours on one machine.
• Native output to WebDataset, TFRecord, Parquet, or plain subfolders.
• Respects robots opt-out directives (noai, noimageai) by default.
• Distributed downloading via multiprocessing, PySpark, or Ray.
• Sidecar metadata includes EXIF, dimensions, SHA-256, and per-URL success/failure status.

Caveats

• The README warns that hitting the claimed throughput requires a fast DNS resolver; default system DNS may bottleneck you.
• Because it pulls from the open web, failed downloads and malformed images are expected—the tool tracks these in metadata, but you’ll still need to filter shards.

Verdict

Worth a look if you’re assembling large vision or multimodal datasets from existing URL lists. Skip it if your data is already local or lives behind APIs that don’t serve direct image links.