Vibeleaderboard
Index / tool
Visit evalplus.github.io
Category
Developer Tools
Rank
No. 2841Tools index

Previous survey · No. 2590 ·

Pricing
Open Source
Type
TOOL
Use case
Model & Agent Evaluation
Date

About

Code-evaluation framework that adds test inputs to HumanEval and MBPP and shows base and expanded-test scores. The public dataset controls were inspected and relevant loaders and checker source were read without executing candidate code. README and website task-count notes differ, so reproductions need the actual dataset revision and task set.

At a glance

EvalPlus is worth using as a check on coding scores that may depend on too few tests. It extends HumanEval and MBPP with more inputs, so a generated function that passes the original examples can still fail the expanded evaluation. Its public comparison makes that gap visible. Use it to investigate code correctness on small programming tasks, alongside evidence from the actual application you want a model to work on.

Good fit

People comparing code-generation models or checking whether an apparently successful solution survives edge cases. The inspected evaluator runs the same candidate against base inputs and additional inputs, comparing outputs with the reference implementation. The public page shows Base Tests and EvalPlus Tests side by side, and switching HumanEval to MBPP changed the selected dataset and displayed scores. This is more informative than treating one passing example as sufficient evidence of correctness.

Consider something else when

You need proof that code is secure, maintainable or suitable for a large repository. Passing a finite collection of function tests does not establish those properties. The root site also links EvalPerf for code-efficiency evaluation and RepoQA for repository understanding; those answer different questions. A correctness score should not be presented as a speed score, and a historical model row is not a current recommendation for every model with a similar name.

Start here

Open the correctness comparison page and choose HumanEval or MBPP. Compare the same model and generation setup across Base Tests and EvalPlus Tests before using Average. A drop tells you to investigate which additional inputs failed. For reproduction, the package documents evalplus.evaluate with a dataset and either a model backend or an existing samples file. Review the actual dataset revision and the execution environment first. No candidate code or model was run for this guide.

Cost and requirements

The public pages opened without an account, payment or API key. The package declares Apache-2.0 and Python 3.9 or later; model inference and compute remain separate costs, and the declared Python minimum does not establish compatibility for every dependency. Candidate evaluation executes Python, and its own reliability guard explicitly says it is not a security sandbox. Use an appropriate isolated environment for a reproduction. Installation, containers, model downloads and execution were not tested here.

Limits to know

Version and task-selection differences can change a score before the model changes. The current README says MBPP+ 0.2.0 reduced the set from 399 to 378 tasks, while the comparison page still describes 399; record the actual loaded task set rather than assuming those notes match. HumanEval+ and MBPP+ data loaders and the base/plus evaluation paths were inspected, not every test or execution boundary. Contamination, prompt style and sampling also limit what a published result tells you about new work.

Still unclear

  • Which additional tests reveal failures relevant to the reader’s own code?
  • Can a result be reproduced with the same task selection, prompt format, samples and execution limits?

Why it made the leaderboard

Useful for investigating generated functions that pass original examples but fail additional inputs. The guide distinguishes finite-test correctness from security, maintainability and speed, explains version and execution boundaries, and links a related benchmark instead of presenting one score as application readiness.

Tags

benchmarkhumanevalmbppcode-generationevaluationleaderboard

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.