HumanEval
https://github.com/openai/human-eval- Category
- Developer Tools
- Rank
- No. 665Tools index
- Listed in
- #9 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Builder
- openai
- GitHub
- 3.4k stars
- Date
About
OpenAI's code evaluation dataset and harness from the Codex paper — 164 hand-written Python problems for benchmarking LLMs on code.
What it does
HumanEval accepts Python completions in JSON Lines, combines them with task prompts and tests, then executes each candidate under a timeout. It records passed, failed, or timed-out outcomes and calculates pass@k estimates only when every problem has enough samples.
Why it's ranked here
Its appeal is a narrow, inspectable evaluation loop with machine-readable inputs and detailed outputs. Per-completion results support debugging, while aggregate pass@k estimates support comparisons. The serious reservation is execution safety: generated code runs locally, and the included guard explicitly is not a sandbox.
What's good
The runner preserves each completion alongside its outcome, making aggregate results traceable to individual failures. It processes evaluations concurrently, isolates candidates in separate processes and temporary directories, suppresses interactive input and output, enforces time limits, and avoids reporting unsupported pass@k estimates.
Tradeoffs
The documentation says code execution starts disabled, but the supplied source contains active execution. That mismatch makes the safety warning easier to misread. The defensive guard blocks many destructive operations but explicitly provides no security boundary. Low memory can also cause correct programs to fail.
How to use it well
It suits engineers comparing Python code generators with repeated samples across a fixed task set. Generate completions elsewhere, validate the JSON Lines feed, then run evaluation inside a robust external sandbox and inspect both aggregate and per-sample results. It does not generate code or provide general-purpose security isolation.
Technical notes+
setup.py declares the human-eval package, dependencies from requirements.txt, and a console script named evaluate_functional_correctness. human_eval/data.py streams plain or gzip-compressed JSON Lines. human_eval/evaluation.py schedules checks through ThreadPoolExecutor, computes estimate_pass_at_k, and writes an augmented results file. human_eval/execution.py builds a program from the prompt, completion, tests, and entry point, then runs it in a multiprocessing.Process with temporary-directory, I/O suppression, timeout, and reliability_guard controls. Critically, README.md says the execution call is commented out, while human_eval/execution.py contains active exec(check_program, exec_globals) code.
Observed
- License
- MIT License
- Primary language
- Python
- Runtime requirement
- Python 3.7 or later
- Installation
- Editable pip installation from a checked-out repository
- Interfaces
- Command-line evaluator and importable Python library helpers
- Dependencies
- tqdm, fire, and numpy
- Data format
- JSON Lines input and output, with plain and gzip-compressed reading and writing
Read from README.md, setup.py, requirements.txt, human_eval/data.py, human_eval/execution.py, human_eval/evaluation.py, human_eval/evaluate_functional_correctness.py, LICENSE, data/example_problem.jsonl, data/example_samples.jsonl.
What it can do
Evaluate code generation capabilities of language models
Language model and Python programming problems → Performance scores and evaluation metrics
Benchmark LLM coding performance
Large language model outputs on coding tasks → Comparative benchmark results
Test Python code solutions against problem specifications
Generated Python code and problem requirements → Pass/fail results and correctness validation
Provide hand-written Python programming problems
Request for coding evaluation tasks → Set of 164 curated Python programming problems
Run automated code testing harness
Python code solutions → Test execution results and error reports
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.