Karpathy's weekend hack: a chat app where LLMs peer-review each other

What it does

LLM Council is a local web app that acts like a multiplayer ChatGPT. It routes your prompt to several LLMs at once via OpenRouter, collects their individual answers in a tabbed view, then forces every model into a second round where they anonymously review and rank the others’ outputs. A configurable CHAIRMAN_MODEL finally distills the whole debate into a single answer.

The interesting bit

The anonymization during peer review is the clever hook: stripping model identities from the responses before grading means (in theory) less brand loyalty and more honest critique. Karpathy also openly admits the whole thing was “99% vibe coded” in a Saturday afternoon as a toy for reading books with AI, which makes the bureaucratic three-stage pipeline feel almost like performance art.

Key highlights

• Queries multiple models in parallel and surfaces every response in a tabbed UI for side-by-side inspection
• Anonymizes outputs during the review stage so models judge answers without knowing which rival produced them
• Designates a configurable CHAIRMAN_MODEL to compile the ranked opinions into a final synthesized response
• Persists conversations as plain JSON files in a local directory
• Explicitly unsupported: the author calls it ephemeral inspiration and suggests asking an LLM to patch it if needed

Caveats

• Requires an OpenRouter API key and purchased credits; it is not free or fully offline
• The example model names in the README (e.g., gpt-5.1, claude-sonnet-4.5) appear to be speculative placeholders, so the default config may not match currently available endpoints
• The author states there is no support, no roadmap, and no intention to improve the project

Verdict

Worth a spin if you want to see how different models reason about the same hard question, but avoid it if you need a maintained product or are trying to dodge API bills.