A Swiss Army fetcher for the AI agent era

What it does

x-reader is a Python content fetcher that takes a URL from YouTube, X, WeChat, Bilibili, Telegram, RSS, or generic web pages and returns structured data in a single UnifiedContent schema. It handles text extraction, video subtitles, and audio transcription through platform-specific fetchers and escalating fallbacks. The project packages this core into three forms: a CLI/library, an MCP server for AI agents, and optional Claude Code skills for deeper transcription and analysis.

The interesting bit

The real craft is in the fallback chains. For X alone, it tries oEmbed, then FxTwitter, then Jina Reader, then a local Playwright session if you are logged in—each step escalating in fidelity and invasiveness. The author treats “universal reader” as an integration problem, not a scraping problem, leaning on existing tools like Jina Reader, yt-dlp, and Groq Whisper rather than reinventing parsers.

Key highlights

• Unified UnifiedContent schema normalizes articles, tweets, videos, and podcasts into one data model.
• Three composable layers: Python CLI/library, Claude Code skills for transcription and analysis, and an MCP server.
• Platform-specific fetchers with graceful degradation, including headless Playwright fallbacks for anti-scraping sites like WeChat and Xiaohongshu.
• Local session management for gated content; cookies stay local unless explicitly allowed.
• Optional dual output to JSON and Markdown, including direct Obsidian vault integration.

Caveats

• The MCP server and Claude Code skills require cloning the repository; they are not shipped in the PyPI install.
• Several transcription and fallback features depend on external services and API keys (Groq for Whisper, Jina Reader, Telegram API credentials).
• Anti-scraping fallbacks require installing a headless browser and performing one-time manual logins for platforms like X and Xiaohongshu.

Verdict

Developers building AI agents, knowledge pipelines, or personal archiving tools will find this a pragmatic time-saver. If you only need to scrape one static site, it is probably over-engineered.