LLM Council
https://github.com/karpathy/llm-council- Category
- AI Agents
- Rank
- No. 649Tools index
- Pricing
- Open Source
- Type
- AGENT
- Builder
- karpathy
- GitHub
- 24.7k stars
- Date
About
Andrej Karpathy's experiment: an ensemble of LLMs debating your hardest questions and arriving at a synthesized answer.
What it does
LLM Council is a local chat application that sends one prompt to several models through OpenRouter. It displays their separate answers, asks the models to assess anonymized peers, calculates aggregate placements, then has a designated chairman produce the response you read last.
Why it's ranked here
Its strongest case is transparency: users can inspect individual answers, raw peer evaluations, parsed placements, and the final synthesis. Parallel requests keep each collection phase practical. However, the author explicitly describes it as an unsupported experiment, so the architecture is more useful for exploration and adaptation than dependable production deployment.
What's good
Anonymized peer assessment is a concrete attempt to reduce favoritism toward named providers. Failed model requests are omitted while successful answers continue through the process. The interface exposes intermediate work instead of hiding it behind the final answer, making disagreements and parsing errors easier to spot.
Tradeoffs
Every message requires repeated paid OpenRouter requests across sequential stages, so the design adds cost and waiting compared with asking one model. Models are configured in code, conversations use local JSON files, and evaluation metadata is not persisted. The author promises no support or future improvement.
How to use it well
Use it for exploratory questions where comparing reasoning, disagreements, and peer judgments matters, including assisted reading or model evaluation. Inspect the source answers and evaluations before trusting the synthesis. It does not replace a maintained hosted service, durable analytics system, or production conversation platform.
Technical notes+
backend/main.py defines the FastAPI service, conversation endpoints, and an SSE streaming route on port 8001. backend/council.py runs parallel first responses, anonymizes them as lettered labels, parses free-text rankings with regular expressions, averages recorded positions, and sends all responses plus evaluations to the chairman. backend/openrouter.py uses async httpx requests and converts request failures to None. backend/storage.py writes one JSON document per conversation under data/conversations/, but ranking metadata returned by the API is not saved. frontend/src/api.js targets the fixed local backend URL and parses streamed events. backend/config.py hardcodes council members, the chairman, provider endpoint, and storage directory.
Observed
- Languages
- Python backend with JavaScript and JSX frontend
- Install surface
- uv manages Python dependencies; npm manages frontend dependencies
- Interface
- Local React and Vite web application backed by a FastAPI HTTP API
- Provider requirement
- Requires an OpenRouter API key and provider credits
- Storage
- Conversations are stored as local JSON files
- Runtime requirement
- Python 3.10 or newer
Read from README.md, pyproject.toml, main.py, backend/main.py, backend/config.py, backend/council.py, backend/storage.py, backend/__init__.py, backend/openrouter.py, frontend/vite.config.js, frontend/eslint.config.js, frontend/src/api.js, uv.lock, start.sh, CLAUDE.md.
What it can do
Facilitate multi-LLM debate on complex questions
Difficult or complex question from user → Multiple LLM perspectives and arguments
Synthesize consensus answer from LLM ensemble
Debate outputs from multiple LLMs → Single synthesized answer or conclusion
Compare different LLM reasoning approaches
Question or problem statement → Side-by-side comparison of LLM reasoning methods
Generate diverse perspectives on controversial topics
Contentious or multi-faceted question → Multiple viewpoints and reasoning chains
Validate answers through cross-LLM verification
Initial answer from one LLM → Verified or challenged answer with supporting evidence
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.