Sunsetted MCP server that made your IDE test its own web apps

What it does

web-eval-agent was an MCP server that plugged into editors like Cursor, Cline, and Windsurf to autonomously test running web apps. You gave it a URL and a plain-English task—“run through the signup flow and report UX issues”—and it spun up a Playwright browser, clicked through the UI, captured screenshots, console logs, and network traffic, then delivered a structured report back into your IDE chat. The project has been officially discontinued; the team is now building something new at withrefresh.com.

The interesting bit

The recursive appeal: your coding agent could invoke another agent to verify that the code it just wrote actually worked in a real browser. It also handled authentication state by popping up an interactive browser window for manual login, then reusing those cookies in subsequent headless runs.

Key highlights

• Two MCP tools: web_eval_agent for autonomous testing and setup_browser_state for interactive login sessions
• Captured and filtered network traffic, console errors, and screenshots into a chronological timeline
• Required an API key from operative.sh and used the BrowserUse framework with an operative backend
• Returned structured natural-language reports with step-by-step agent actions and UX judgments
• Targeted “vibe-coding” workflows where the IDE writes and verifies code without human intervention

Caveats

• Discontinued: The project is officially sunsetted and no longer maintained
• External dependency: Required a free API key from operative.sh and relied on their backend infrastructure
• Platform rough edges: Windows manual setup was explicitly noted as unfinished at the time of sunset

Verdict

Worth a look if you’re researching MCP-based QA automation or “vibe-testing” patterns, but don’t build production workflows on a dead repo. If you need a living tool, follow the team to withrefresh.com or look elsewhere.