GPQA Diamond
github.com/idavidrein/gpqa- Category
- Developer Tools
- Rank
- No. 3300Tools index
Previous survey · No. 3164 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- CLI
- GitHub
- 542 stars
- Date
About
Graduate-level science questions in biology, chemistry, and physics. Diamond is the expert-validated subset used in many model reports.
What it does
GPQA is a set of hard four-option science questions built to be Google-proof: a determined non-expert with a search engine should still struggle. The repository is the companion code for the paper. It distributes the data as a password-protected archive, with a Hugging Face mirror, and adds a Python harness that runs OpenAI models against it in several styles: plain zero-shot, five worked examples, step-by-step reasoning, a majority vote over repeated samples, and an open-book mode where the model writes its own Bing queries and can read a scraped page.
Why it's ranked here
The questions matter more than the code. GPQA is a peer-reviewed benchmark (published at the Conference on Language Modeling), MIT licensed, and its Diamond split is the figure many model reports quote. The data carries a canary string that extends the BigBench one, so contamination filters can find leaked copies. The harness is a research artifact rather than a maintained tool: it knows a short, fixed list of older models and has no tests, so its value is reproducing the paper's baselines, not scoring new systems.
What's good
Answer order is shuffled with a seed you choose, so position bias can be varied and any run reproduced exactly. Every model response is cached per model and prompt, so an interrupted or repeated run costs nothing extra. Each run writes a per-question CSV log holding the question, the correct answer, the chosen answer and the full model response, and responses with no parseable letter are counted as refusals rather than dropped silently. The open-book mode caps itself at six searches, or three when it also scrapes pages, which bounds cost per question.
Tradeoffs
Only a hard-coded list of models works; any other name raises an error, and the README names just two OpenAI chat models. The open-book module reads the Bing key from the environment when it is imported, and the main runner always imports it, so even closed-book runs fail without that variable set. One question position is hard-coded as a failure with a comment calling it a hack for long answers. Answer parsing falls back to the first bracketed letter anywhere in the reply. The README lists a few-shot mode that the code rejects in favour of a five-shot one. No test directory exists in the tree.
How to use it well
Treat this as the source of the data and the reference for how the paper scored models. Pull the questions from Hugging Face, point the runner at the split you want, and cap the sample count on a first pass to keep API spend small. Set the Bing variable, even to a placeholder, before closed-book runs. To evaluate a current model, reuse the question files and the seeded shuffle in your own harness, since this one only speaks to older APIs. Keep the questions out of public repositories and pages; the canary string exists because leakage ruins the benchmark.
Technical notes+
baselines/run_baseline.py exposes the AnswerPredictor class through Python Fire; responses are pickled to cache_{model}.pkl keyed on (model, prompt_type, prompt), self-consistency draws SELF_CONSISTENCY_SAMPLES = 20 at temperature 0.7, question_id 69 is short-circuited as a wrong answer, and parse_sampled_answer tries four phrasings before a bare \((.)\) regex. Accuracy divides by the full example count, so skipped and unparsed items count as wrong. baselines/utils.py holds CHAT_MODELS, OPENAI_MODELS and ANTHROPIC_MODELS (only claude-2, on the legacy completion interface), MAX_NUM_RETRIES = 5, and load_examples, which reads the CSV with pandas and shuffles the four choices after random.seed(seed). The five-shot and chain-of-thought prompts load a JSON examples file from the prompts directory; zero-shot chain of thought generates reasoning traces up front and caches them separately. baselines/open_book.py reads BING_SEARCH_V7_SUBSCRIPTION_KEY via os.environ at import, leaves the Google path as a stub returning a placeholder string, reads OpenAI-shaped responses (a TODO notes Anthropic is unimplemented), and trims scraped pages to 2000 characters either side of the sentence with the best word-overlap F1 against the search snippet. prompts/chain_of_thought.txt holds five worked examples across genetics, chemistry, quantum mechanics, organic chemistry and probability; the titration explanation repeats two lines. requirements.txt pins exact versions of the openai and anthropic SDKs alongside transformers, pandas and wandb.
Observed
- License
- MIT (LICENSE)
- Language
- Python, dependencies pinned in requirements.txt
- Interface
- Command-line runner built on Python Fire (baselines/run_baseline.py)
- Supported model APIs
- Hard-coded OpenAI completion and chat models plus one Anthropic legacy-completion model (baselines/utils.py)
- Data distribution
- Password-protected archive in the repository, mirrored as a Hugging Face dataset
- Contamination guard
- Canary string in the data, a superset of the BigBench canary
- Open-book dependency
- Bing Web Search API key read from the environment at import (baselines/open_book.py)
- Tests
- No test directory in the repository tree
Read from README.md, requirements.txt, baselines/run_baseline.py, baselines/utils.py, baselines/open_book.py, prompts/chain_of_thought.txt, LICENSE.
What it can do
Run zero-shot and few-shot baseline evaluations of GPT-3.5/GPT-4 on GPQA questions
GPQA questions → Model answers/accuracy scores
Run chain-of-thought prompting evaluation on GPQA questions
GPQA questions → Model answers with reasoning
Run open-book retrieval-augmented evaluation using Bing search
GPQA questions → Model answers using retrieved web content
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.