A self-hosted scraper that speaks fluent LLM

What it does

WaterCrawl is a web crawling platform built on Django, Scrapy, and Celery that you host yourself. It crawls and scrapes sites with configurable depth and speed, then serves the results through a REST API with OpenAPI documentation. A built-in search engine offers tiered discovery modes—basic, advanced, and ultimate—with multi-language and country-specific targeting. Real-time job progress streams back via Server-Sent Events.

The interesting bit

Instead of treating scraping as a scripting chore, WaterCrawl wraps the mature Python scraping stack into an API-first service with SDKs and no-code integrations. It explicitly targets LLM and automation pipelines, offering plugins for OpenAI and hookups for Dify and N8N.

Key highlights

• Self-hosted stack: Django, Scrapy, Celery, and Docker-based deployment
• REST API with OpenAPI spec and SSE-based real-time progress updates
• Tiered search engine with basic, advanced, and ultimate depth levels
• Client SDKs for Python, Node.js, Go, and PHP
• Integrations for Dify, N8N, and an OpenAI plugin

Caveats

• Non-localhost deployments require manual MinIO endpoint configuration; the README warns that skipping this breaks file uploads and downloads
• Production setup demands additional database and service configuration beyond the default Docker Compose environment
• The Langflow integration is currently an unmerged pull request

Verdict

Best for teams that need a private, API-driven crawling backend for RAG or automation. Not for anyone looking for a fully managed SaaS or a lightweight CLI tool.