EvalPlus
evalplus.github.io- Category
- Developer Tools
- Rank
- No. 2841Tools index
Previous survey · No. 2590 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Date
About
Code-evaluation framework that adds test inputs to HumanEval and MBPP and shows base and expanded-test scores. The public dataset controls were inspected and relevant loaders and checker source were read without executing candidate code. README and website task-count notes differ, so reproductions need the actual dataset revision and task set.
At a glance
EvalPlus is worth using as a check on coding scores that may depend on too few tests. It extends HumanEval and MBPP with more inputs, so a generated function that passes the original examples can still fail the expanded evaluation. Its public comparison makes that gap visible. Use it to investigate code correctness on small programming tasks, alongside evidence from the actual application you want a model to work on.
Good fit
People comparing code-generation models or checking whether an apparently successful solution survives edge cases. The inspected evaluator runs the same candidate against base inputs and additional inputs, comparing outputs with the reference implementation. The public page shows Base Tests and EvalPlus Tests side by side, and switching HumanEval to MBPP changed the selected dataset and displayed scores. This is more informative than treating one passing example as sufficient evidence of correctness.
Consider something else when
You need proof that code is secure, maintainable or suitable for a large repository. Passing a finite collection of function tests does not establish those properties. The root site also links EvalPerf for code-efficiency evaluation and RepoQA for repository understanding; those answer different questions. A correctness score should not be presented as a speed score, and a historical model row is not a current recommendation for every model with a similar name.
Start here
Open the correctness comparison page and choose HumanEval or MBPP. Compare the same model and generation setup across Base Tests and EvalPlus Tests before using Average. A drop tells you to investigate which additional inputs failed. For reproduction, the package documents evalplus.evaluate with a dataset and either a model backend or an existing samples file. Review the actual dataset revision and the execution environment first. No candidate code or model was run for this guide.
Cost and requirements
The public pages opened without an account, payment or API key. The package declares Apache-2.0 and Python 3.9 or later; model inference and compute remain separate costs, and the declared Python minimum does not establish compatibility for every dependency. Candidate evaluation executes Python, and its own reliability guard explicitly says it is not a security sandbox. Use an appropriate isolated environment for a reproduction. Installation, containers, model downloads and execution were not tested here.
Limits to know
Version and task-selection differences can change a score before the model changes. The current README says MBPP+ 0.2.0 reduced the set from 399 to 378 tasks, while the comparison page still describes 399; record the actual loaded task set rather than assuming those notes match. HumanEval+ and MBPP+ data loaders and the base/plus evaluation paths were inspected, not every test or execution boundary. Contamination, prompt style and sampling also limit what a published result tells you about new work.
Related work
Still unclear
- Which additional tests reveal failures relevant to the reader’s own code?
- Can a result be reproduced with the same task selection, prompt format, samples and execution limits?
Sources checked 2026-10-31. Source-based assessment.
- evalplus.github.io
- evalplus.github.io/leaderboard.html
- evalplus.github.io/evalperf.html
- github.com/evalplus/evalplus/blob/master/README.md
- github.com/evalplus/evalplus/blob/master/LICENSE
- github.com/evalplus/evalplus/blob/master/setup.cfg
- github.com/evalplus/evalplus/blob/master/evalplus/evaluate.py
- github.com/evalplus/evalplus/blob/master/evalplus/eval/__init__.py
- github.com/evalplus/evalplus/blob/master/evalplus/eval/utils.py
- github.com/evalplus/evalplus/blob/master/evalplus/data/humaneval.py
- github.com/evalplus/evalplus/blob/master/evalplus/data/mbpp.py
Why it made the leaderboard
Useful for investigating generated functions that pass original examples but fail additional inputs. The guide distinguishes finite-test correctness from security, maintainability and speed, explains version and execution boundaries, and links a related benchmark instead of presenting one score as application readiness.
Tags
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.