Vibeleaderboard
Index / tool

Humanity's Last Exam

lastexam.ai
Visit lastexam.ai
Category
Developer Tools
Rank
No. 1063Tools index

Previous survey · No. 1024 ·

Pricing
Free
Type
TOOL
Use case
Model & Agent Evaluation
Date

About

A closed-ended academic benchmark built to outlast frontier models: 2,500 questions across more than 100 subjects, written by close to 1,000 expert contributors at over 500 institutions in 50 countries, and published in Nature in January 2026. Frontier scores stay low — Gemini 3 Pro at 38.3%, GPT-5 at 25.3% — and a private held-out set is kept alongside the public HuggingFace dataset to detect overfitting to the released questions.

What it does

Humanity's Last Exam is a question bank plus a small grading kit. The questions, some with images, live on Hugging Face and are either multiple choice or short answer with one exact correct answer. The repository supplies two scripts: one sends every question to a model and saves its explanation, answer and stated confidence; the other asks a second model to compare each saved answer with the official one, then prints accuracy with a 95% error bar and a calibration error that shows whether the model's confidence tracks how often it is actually right.

Why it's ranked here

It is the reference exam for frontier reasoning claims, and the repository backs that up with method rather than marketing. The paper is in Nature, the code is MIT licensed, the questions ship with a canary string so labs can filter them out of training data, and a public change log records every question added, removed or corrected by ID. The tool-use protocol for the harder Diamond subset is written down in detail, down to which web tools a model may call and what must be blocked. The harness itself is thin, but the benchmark is the product.

What's good

Grading is mechanical and repeatable. The judge model returns a fixed structured verdict per question: the extracted answer, its reasoning, a yes or no, and the confidence the tested model claimed. Both scripts cache results to disk and skip finished questions on rerun, so a run that hits rate limits resumes instead of starting over. Calibration is reported next to accuracy, which matters on an exam where most answers are wrong: a model that is confidently wrong scores badly twice. The tool-enabled protocol pins harness versions per model and spells out search-result and page filtering, so an agentic score has stated conditions.

Tradeoffs

The kit speaks only the OpenAI chat interface, so other providers need a compatible endpoint. The temperature option is accepted on the command line but never sent to the model, so changing it does nothing. The question set moves: the change log shows batches of removals, re-additions and new questions, and the README's sample run grades a different total than the question count it states, so scores from different dates are not directly comparable. There is no test directory, three pinned dependencies make up the whole install, and the default judge is itself an OpenAI model, so the grade inherits that judge's mistakes.

How to use it well

Use it to check a vendor's reasoning claim or to track your own model across versions, not to compare one model against another casually. Record which version of the question set you ran and which judge model did the judging, since both move the number. Try a handful of questions first, then a full run, and rerun to pick up failed calls. Give reasoning models a generous output limit: the authors report collapse below about 8,000 tokens. For agent scores, follow the published tool protocol exactly. It measures closed-ended expert knowledge, not coding, long tasks or open-ended writing.

Technical notes+

hle_eval/run_model_predictions.py loads the cais/hle test split with the datasets library, wraps each question (plus an image_url part when the image field is non-empty) in a fixed SYSTEM_PROMPT demanding Explanation / Answer / Confidence, and fans out through AsyncOpenAI (timeout 600s, max_retries=1) under an asyncio.Semaphore of --num_workers; it sends the system prompt as a user message when the model name contains "o1". The temperature argument is parsed but its line in the chat.completions.create call is commented out. Results cache to hle_<model>.json keyed by question id. hle_eval/run_judge_results.py calls client.beta.chat.completions.parse with the ExtractedAnswer pydantic schema, default judge o3-mini-2025-01-31, writes judged_<predictions>.json, and in dump_metrics divides correct answers by the full question count (so unanswered questions count as wrong), reports a Wald 95% half-width, and computes RMS calibration error over confidence-sorted bins of 100 adapted from hendrycks/outlier-exposure. requirements.txt pins numpy 1.26.4, openai 1.57.0 and datasets 3.2.0. docs/evaluation-with-tools.md defines the HLE-Diamond tool setting: no sandbox internet, only web_search (20 results max) and web_fetch (60,000 characters max), a JSON blocklist applied to queries, results and fetched pages, and per-model harness versions. hle-rolling-changes.txt logs add, remove, readd and update entries by question id.

Observed

License
MIT (evaluation code)
Language
Python
Install surface
pip install of three pinned packages: numpy, openai, datasets
Interface
Two command-line scripts: prediction run and judge run
Model API
OpenAI Python client only; judge uses structured output parsing
Data distribution
Questions hosted on Hugging Face as cais/hle, not in the repository
Metrics reported
Accuracy with Wald confidence interval, RMS calibration error
Contamination guard
Canary GUID string for filtering from training corpora
Question set changes
Public per-question change log of adds, removals and updates
Tests
No test directory in the repository tree
Publication
Nature, volume 649, with arXiv preprint 2501.14249

Read from README.md, requirements.txt, LICENSE, hle_eval/run_model_predictions.py, hle_eval/run_judge_results.py, docs/evaluation-with-tools.md, hle-rolling-changes.txt, citation.txt.

Tags

benchmarkreasoningfrontier-modelsevaluationleaderboardacademic

Tech Stack

Python

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.