egoist/sitefetch
A CLI tool and library that crawls websites and extracts readable text content for consumption by AI models.
Not currently ranked — collecting fresh signals.
star history
Sitefetch is a web scraping tool that recursively fetches an entire website and saves the content as plain text files. It uses Mozilla’s Readability library to extract clean article content from pages, supports path matching with glob patterns to target specific pages, and allows CSS selectors for precise content extraction. The extracted text can be fed directly into LLMs for tasks like RAG, analysis, or building context from web sources.
Frequently asked
- What is egoist/sitefetch?
- A CLI tool and library that crawls websites and extracts readable text content for consumption by AI models.
- Is sitefetch open source?
- Yes — egoist/sitefetch is open source, released under the MIT license.
- What language is sitefetch written in?
- egoist/sitefetch is primarily written in TypeScript.
- How popular is sitefetch?
- egoist/sitefetch has 1.7k stars on GitHub.
- Where can I find sitefetch?
- egoist/sitefetch is on GitHub at https://github.com/egoist/sitefetch.