The web's content sausage machine, backed by actual linguistics

What it does Trafilatura fetches web pages, strips the boilerplate, and returns the actual article text plus metadata—title, author, date, tags, even comments if you want them. It outputs to CSV, JSON, Markdown, XML-TEI, or plain text. No database required; it runs as a Python library or standalone CLI tool.

The interesting bit This isn’t another weekend scraper project. It grew out of a linguistics PhD and a research institute’s need for clean web corpora. The name means “wire drawing” in Italian—the process of pulling metal through dies to refine it. The extraction logic reflects that pedigree: it balances precision and recall using a mix of pattern matching and algorithms like jusText and readability, then validates itself against academic benchmarks.

Key highlights

• Benchmarked as top performer in multiple independent evaluations, including ScrapingHub’s article extraction benchmark and a 2023 ROUGE-LSum comparison
• Handles the full pipeline: sitemap/feed discovery, polite crawling with deduplication, extraction, and formatting
• Preserves document structure—paragraphs, lists, quotes, code blocks, tables—rather than flattening everything to text soup
• Optional language detection and speed optimizations
• Used by HuggingFace, IBM, Microsoft Research, Stanford, and thousands of downstream projects

Caveats

• The README notes the project’s “future depends on community support” and explicitly asks for sponsors—a signal that maintenance bandwidth may be constrained
• Prior to v1.8.0 the license was GPLv3+; current versions are Apache 2.0, so check your dependency tree if license purity matters

Verdict If you’re building RAG pipelines, training LLMs, or doing quantitative text analysis, this saves you from writing your 47th ad-hoc HTML cleaner. If you just need to scrape one page once, it’s probably overkill—but you’ll still get better results than regex and prayers.