A Go scraper that escalates from curl to fake Firefox to paid API

What it does

Anakin is a self-hosted web scraping API written in Go. It fetches pages, strips boilerplate, and returns clean markdown or structured JSON via an optional Gemini integration. It ships as a single ~30MB binary that needs no database, Redis, or message queue to get started.

The interesting bit

Every scrape walks a handler chain: a fast HTTP request first, then Camoufox—an anti-detect Firefox with realistic fingerprints—if that fails, and only then a paid external API. The proxy pool uses Thompson Sampling, a multi-armed bandit algorithm, to learn per-domain which proxy is actually working instead of rotating blindly.

Key highlights

• Single Go binary; zero-config mode stores jobs in memory with no Docker or PostgreSQL required
• Handler chain fallback (HTTP → Camoufox browser → external API) keeps paid API calls to roughly the ~5% of pages that actually need them
• Anti-detect browser uses Camoufox rather than headless Chrome, connecting over WebSocket via Playwright
• Built-in React dashboard for scraping, job tracking, domain configs, and live proxy score monitoring
• Optional Gemini-powered structured JSON extraction (bring your own API key)

Caveats

• In-memory storage means job history vanishes on restart unless you wire up PostgreSQL
• JavaScript-heavy sites require the optional browser service in Docker; the bare binary handles static pages but not SPAs
• Structured JSON extraction requires a GEMINI_API_KEY; the core scraper needs no API keys, but the AI feature is strictly BYOK

Verdict

Ideal if you want clean, LLM-ready markup without shipping data to another cloud service. Skip it if you are looking for a fully managed SaaS and do not mind paying per page regardless of difficulty.