Your agent's browser, minus the ads and nav bars

What it does

webclaw is a Rust-based extraction toolkit that fetches web pages and distills them into clean markdown, plain text, JSON, or an LLM-optimized format. It works as a local CLI, an MCP server that plugs into Claude or Cursor, or a self-hostable REST API. The core engine strips navigation, ads, scripts, and boilerplate so downstream tools get readable context instead of raw markup.

The interesting bit

The project is built around a “local-first” split: core scraping, crawling, batching, and even LLM-powered summarization can run entirely offline using local providers like Ollama. Only search, research workflows, and JavaScript rendering force a trip to the hosted API. That keeps sensitive or high-volume jobs off someone else’s infrastructure without sacrificing a managed tier when you need it.

Key highlights

• Ships as a CLI, MCP server, and REST API with SDKs for TypeScript, Python, and Go.
• Core extraction logic (webclaw-core) is decoupled from network I/O, so it can be reused independently.
• Supports same-origin crawling, batch parallel scraping, page diffs, and brand-asset extraction (colors, fonts, logos).
• Can route through proxy pools and offers a Firecrawl-compatible API example for drop-in migration.
• Local LLM features (summarization, structured extraction) work via Ollama or OpenAI-compatible endpoints.

Caveats

• JavaScript rendering, web search, and multi-source research workflows are gated behind the hosted API, so fully air-gapped use is limited to static pages.
• The project openly solicits failing fixtures for “messy HTML,” which suggests extraction heuristics are still a work in progress on irregular sites.

Verdict

Worth a look if you’re building RAG pipelines or AI agents that need clean web context without managing a browser farm. Skip it if you need heavy JavaScript SPAs or large-scale search entirely offline.