LiveCodeBench
livecodebench.github.io- Category
- Developer Tools
- Rank
- No. 1941Tools index
Previous survey · No. 1836 ·
- Listed in
- #6 Find AI benchmarks
- Type
- TOOL
- Builder
- livecodebench
- GitHub
- 941 stars
- Date
About
Contamination-aware coding benchmark that continuously collects new contest problems and measures code generation, repair, and execution.
What it does
LiveCodeBench runs language models against versioned programming-problem sets, formats prompts for several model families, executes generated answers, and reports pass-based metrics. Separate scenarios exercise code writing, self-repair, output prediction, and code execution. It can query hosted models or run local models through vLLM.
Why it's ranked here
This is a strong evaluation harness because it combines reproducible dataset releases, multiple coding tasks, local and hosted inference, resumable runs, and per-problem result artifacts. Time-window filtering gives teams a practical way to compare models on problems published after a chosen cutoff. The main reservation is operational weight and some documented measurement variance from execution timeouts.
What's good
Versioned datasets make comparisons repeatable while still allowing newer problem sets. Cached generations and continuation modes reduce wasted inference work. The custom-output evaluator separates generation from grading, which helps teams test outputs produced elsewhere. Prompt routing supports distinct model families, while multiprocessing handles hosted requests and vLLM distributes local inference across available GPUs.
Tradeoffs
Installation pulls heavyweight dependencies including PyTorch, vLLM, dataset tooling, and several provider SDKs. The default fast dataset prunes many tests for speed, so users must opt into the fuller evaluation. Reported metrics can shift slightly with time limits, requiring process-count or timeout tuning. Self-repair depends on earlier code-generation evaluation artifacts and supports only one repair sample per attempt.
How to use it well
Use it for research or model-selection pipelines that need repeatable coding evaluations across several capabilities. Pin a dataset release, record sampling settings, cache generations, and retain detailed evaluation artifacts. Filter by problem publication window when training-data exposure matters, and rerun sensitive comparisons with conservative timeout settings. It does not replace application-specific acceptance tests, production monitoring, or broad software-engineering evaluation beyond contest-style tasks.
Technical notes+
pyproject.toml defines a Python package named livecodebench, requires Python 3.10 or newer, and declares PyTorch, vLLM, provider SDKs, datasets, and Pebble as dependencies. lcb_runner/runner/main.py coordinates benchmark loading, generation, continuation, evaluation, and JSON output. lcb_runner/runner/parser.py exposes the command-line configuration surface. lcb_runner/runner/vllm_runner.py batches local inference, while lcb_runner/runner/oai_runner.py and lcb_runner/runner/grok_runner.py use OpenAI-compatible chat clients. lcb_runner/runner/base_runner.py implements caching and multiprocessing. lcb_runner/utils/scenarios.py defines the four supported scenarios, and lcb_runner/evaluation/__init__.py exports their metric entry points.
Observed
- Primary language
- Python
- Python requirement
- Python 3.10 or newer
- Packaging
- Setuptools project with an editable installation workflow using uv
- Interface
- Command-line runner invoked as a Python module, plus a custom-output evaluation command
- Inference surfaces
- Local inference through vLLM and hosted inference through provider SDKs
- Evaluation scenarios
- Code generation, self-repair, test-output prediction, and code execution
Read from README.md, pyproject.toml, lcb_runner/lm_styles.py, lcb_runner/runner/main.py, lcb_runner/runner/parser.py, lcb_runner/utils/scenarios.py, lcb_runner/prompts/__init__.py, lcb_runner/utils/path_utils.py, lcb_runner/runner/oai_runner.py, lcb_runner/runner/base_runner.py, lcb_runner/runner/grok_runner.py, lcb_runner/runner/vllm_runner.py, lcb_runner/utils/multiprocess.py, lcb_runner/evaluation/__init__.py, lcb_runner/prompts/self_repair.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.