BullshitBench
github.com/petergpt/bullshit-benchmark- Category
- AI Tools
- Rank
- No. 431Tools index
- Listed in
- #76 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- CLI · Web
- Builder
- @petergpt
- GitHub
- 1.9k stars
- Date
About
A benchmark tool that tests whether AI models can detect and challenge nonsensical prompts instead of confidently answering invalid questions. It evaluates models across multiple domains using 100 carefully crafted nonsense questions.
What it does
BullshitBench runs stateless prompts against language models, then asks a fixed three-judge panel to classify each response as clear pushback, partial challenge, or acceptance. Its domain and technique breakdowns expose where a model’s skepticism fails, while the viewer compares reasoning settings, release timing, token use, cost, and model size.
Why it's ranked here
The design goes beyond a prompt collection. It separates collection, judging, aggregation, publishing, and visualization, with explicit consensus rules and refusal handling. That makes results easier to inspect and reproduce. The main caution is judge dependence: canonical outcomes come from three language models using a fixed rubric, not human labels.
What's good
The question set spans software, finance, legal, medical, and physics, with thirteen documented nonsense patterns. Every response receives all three canonical judgments, then mean aggregation. Collection supports per-model concurrency limits, rate-limit requeuing, cooldowns, retries, shuffled tasks, and durable checkpoints. Published artifacts include row-level usage, cache, cost, and throughput fields.
Tradeoffs
Running the benchmark requires provider credentials and potentially many paid requests. Results inherit the judgment panel’s interpretation of the rubric. The prompts are intentionally nonsensical, so this does not measure factual accuracy, ordinary helpfulness, or performance on valid questions. Refusals are tracked separately and excluded from scored records, which readers must consider when comparing detection rates.
How to use it well
Use it when selecting or regression-testing models for assistants that must challenge broken premises. Compare domain mixes and reasoning settings, inspect partial challenges, and retain published artifacts for audit. Tune concurrency and cooldown controls for large runs. Pair it with evaluations for factuality, valid-task completion, safety, latency, and production reliability, because BullshitBench does not cover those needs.
Technical notes+
scripts/openrouter_benchmark.py implements the Python CLI and defines collection, grading, panel aggregation, reporting, provider routing, structured judge output, reasoning levels, and refusal semantics. docs/TECHNICAL.md documents a collect, grade, optional panel, and publish pipeline with exactly three judges, full-panel coverage, and mean consensus. scripts/build_questions_v2_from_draft.py parses legacy difficulty sections and technique-coded Markdown into pipeline-ready JSON. scripts/model_launch_pipeline.py builds model inventory and provenance datasets. scripts/push_bullshitbench_to_forge.py converts aggregate rows into pairwise feedback events. tests/test_refusal_outcomes.py verifies reasoning normalization, refusal classification, score exclusion, and skipped grading for provider refusals.
Observed
- License
- MIT License
- Primary language
- Python
- Interfaces
- Command-line benchmark pipeline and static interactive web viewer
- Provider support
- OpenRouter by default, with per-model routing to OpenAI
- Test structure
- The repository includes unittest coverage for refusal outcomes and reasoning normalization
- Published data surface
- JSONL, JSON, and CSV artifacts for responses, aggregates, summaries, leaderboards, launch metadata, and model parameters
Read from README.md, docs/TECHNICAL.md, scripts/openrouter_benchmark.py, scripts/model_launch_pipeline.py, scripts/push_bullshitbench_to_forge.py, scripts/render_star_milestone_chart.py, scripts/build_questions_v2_from_draft.py, scripts/annotate_star_history_milestone.py, tests/test_refusal_outcomes.py, LICENSE.
What it can do
Test AI model's ability to detect nonsensical prompts
AI model and benchmark dataset → Detection accuracy scores and metrics
Evaluate AI model responses to invalid questions
AI model responses to 100 nonsense questions → Performance evaluation results
Measure AI model confidence levels on nonsense prompts
AI model responses with confidence indicators → Confidence calibration metrics
Compare multiple AI models' nonsense detection performance
Response data from multiple AI models → Comparative performance analysis
Benchmark AI models across 5 different domains
AI model responses to domain-specific nonsense questions → Domain-wise performance breakdown
Generate visual comparisons of model performance
Model benchmark results and scores → Charts and graphs showing performance comparisons
Tags
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.