UltraEval
https://github.com/openbmb/ultraeval- Category
- AI Tools
- Rank
- No. 1620Tools index
Previous survey · No. 1626 ·
- Listed in
- #8 Evaluate an LLM application
- Pricing
- Open Source
- Type
- TOOL
- Builder
- openbmb
- GitHub
- 257 stars
- Date
About
Open-source framework for evaluating foundation models — modular benchmarks for LLM capability assessment (ACL 2024 Demo).
What it does
UltraEval turns selected datasets into repeatable model assessments. You prepare benchmark data, generate a JSON task configuration, expose a model through a local inference endpoint, then run tasks with chosen prompts, metrics, few-shot settings, and generation parameters. It writes task-level metrics, optional instance records, and aggregated dataset results.
Why it's ranked here
The breadth is compelling: the documented benchmark set spans knowledge, mathematics, code, reasoning, reading, translation, similarity, sentiment, and classification. Configurable prompts, metrics, post-processing, and model endpoints make comparisons adaptable. However, the default execution path contains an apparent output-handling bug, and metric failures can silently become zero scores, weakening confidence without extra validation.
What's good
The workflow separates data preparation, model serving, and evaluation, which makes each stage replaceable. It supports local Hugging Face models, personally trained models, vLLM deployment, multiple GPUs, batching, concurrency, few-shot evaluation, and custom datasets. Results retain task configuration, final metrics, optional instance details, and cross-task summaries, giving teams useful material for reproduction and diagnosis.
Tradeoffs
Setup is more involved than a single command: users clone the repository, install all dependencies, download a separate dataset archive, preprocess it, generate configuration, and launch model serving. vLLM is an unconditional package dependency. Several metric implementations catch every exception and return zero, potentially confusing calculation failures with poor model performance. Disabling instance output also appears to leave a printed path unset.
How to use it well
Use UltraEval when an ML team needs configurable, repeatable benchmark runs across local or endpoint-served language models. Start with limited tasks and instance output enabled, inspect raw predictions, then expand coverage after validating metrics and post-processing. It fits comparative capability testing and custom benchmark development. It does not replace production monitoring, application-level quality evaluation, or hosted model operations.
Technical notes+
setup.py packages UltraEval with setuptools, reads requirements.txt wholesale, and requires Python 3.10 or newer. main.py exposes an argparse CLI; Evaluator loads JSON task definitions, seeds Python and NumPy, runs tasks sequentially, prints Markdown tables, and writes aggregated JSON. In Evaluator.run, _save_path is assigned only when args.write_out is true but printed unconditionally, so the default false setting can raise UnboundLocalError. metrics/bleu.py, metrics/chrf.py, and metrics/rouge.py catch broad exceptions and return zero. URLs/vllm_url.py serves vLLM through Flask at POST /infer, supports tensor parallelism from comma-separated GPU IDs, and mutates a process-wide sampling parameter dictionary from request values. utils/utils.py dynamically imports transformation functions from configured paths.
Observed
- License
- Apache-2.0
- Primary language
- Python
- Packaging
- Setuptools package installed with pip from the cloned repository
- Python requirement
- Python 3.10 or newer
- Interfaces
- Command-line evaluation runner and Flask JSON inference endpoint
- Model serving
- Local vLLM serving supports one or multiple specified GPUs
- Dependencies
- Declared packages include vLLM, Flask, OpenAI, SacreBLEU, gevent, Gunicorn, NumPy-related acceleration tooling, and metric utilities
Read from README.md, setup.py, requirements.txt, main.py, docs/tutorials/en/ultraeval.md, docs/tutorials/zh/ultraeval.md, data_process.py, utils/utils.py, metrics/bleu.py, metrics/chrf.py, URLs/vllm_url.py, metrics/rouge.py, utils/request.py.
What it can do
Evaluate foundation model capabilities
Foundation model and benchmark configuration → Performance metrics and assessment scores
Run modular benchmarks on LLMs
Large language model and selected benchmark modules → Benchmark results and capability scores
Compare model performance across benchmarks
Multiple foundation models and benchmark suite → Comparative performance analysis
Configure custom evaluation pipelines
Evaluation parameters and model specifications → Customized benchmark configuration
Generate evaluation reports
Benchmark results and model data → Structured evaluation reports
Assess LLM capabilities across domains
Language model and domain-specific test sets → Domain-specific capability assessments
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.