What it does

LLM Council is a local web app that sends your question to multiple models via OpenRouter, then makes them rank each other’s answers anonymously. A designated “Chairman” model synthesizes the final response. The UI looks like ChatGPT with tabs for inspecting each model’s raw output.

The interesting bit

The anonymization in Stage 2 is the clever nudge: models judge responses without knowing which rival produced them, which theoretically reduces brand loyalty and sycophancy. It’s a lightweight attempt at structured deliberation rather than simple voting or averaging.

Key highlights

• Three-stage pipeline: individual responses → anonymous peer review → chairman synthesis
• FastAPI backend with async httpx, React+Vite frontend, JSON file storage
• Configurable council and chairman models via backend/config.py
• 99% “vibe coded” by Andrej Karpathy as a personal exploration tool
• Requires OpenRouter API key and credits

Caveats

• Author explicitly states zero support: “provided here as is,” “don’t intend to improve it”
• Storage is flat JSON files, not a database
• Cost scales with council size × rounds of API calls

Verdict

Grab it if you want a hackable sandbox for comparing how models critique each other, or if you’re curious about deliberative architectures. Skip if you need a maintained product or are cost-sensitive about running 3+ model calls per query.