- Category
- Developer Tools
- Rank
- No. 936Tools index
Previous survey · No. 909 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- GitHub
- 3.5k stars
- Latest release
- 2.27.14
- Date
About
The standard benchmark for text and multimodal embedding models, covering retrieval, classification, clustering, semantic similarity and reranking rather than a single retrieval score. The 2025 MMTEB expansion extended it to a large multilingual task set, and results are published on an interactive HuggingFace leaderboard. Apache-2.0 and Python, maintained by the embeddings-benchmark organisation.
What it does
A Python toolkit that takes an embedding model, runs it against a library of public datasets, and writes per-task JSON scores you can compare across models. You pick a model by its Hub name, pick tasks or a named suite, and call one evaluate step or one terminal command. Models it does not already know fall back to loading through Sentence Transformers. Beyond text, its docs list model families for images, audio and their pairings with text, so the same harness grades multimodal encoders.
Why it's ranked here
This is the reference harness behind the public embedding leaderboard: getting a model onto that board means adding metadata here, running this package, and opening a results pull request that maintainers review. The package declares itself production stable, ships under Apache 2.0, names three maintainers, and asks users to cite two papers, the 2022 original and the 2025 multilingual expansion. For anyone choosing or shipping an embedding model, it is the shared yardstick rather than one option among several.
What's good
A mock-run command checks a new model wrapper against small built-in tasks that need no dataset downloads, then prints a Markdown pass or fail table, so integration bugs surface in minutes instead of after a long benchmark. Each named suite can return its own citation and export its task list as a LaTeX table for a paper appendix. Results are cached locally and can be pushed as a pull request automatically once the GitHub extra and the GitHub CLI are set up.
Tradeoffs
The core install pulls in PyTorch, Transformers, Sentence Transformers, scikit-learn and Hugging Face datasets, so it is not light. Many model integrations pin conflicting Transformers ranges in their own extras, some below version 5 and some at or above it, so testing several vendors may need separate environments. Python is limited to 3.10 through 3.14. Leaderboard submissions wait on human review, which the docs aim to finish in under a week, and reviewers may question suspicious scores or data leakage.
How to use it well
Use it when you need numbers on an embedding or reranking model that others will trust: comparing candidates for a search feature, or publishing a new model. Start with the mock run before a full suite, keep the local result cache so reruns skip finished tasks, and install only the extra your model needs. It measures embeddings on fixed public tasks; it does not test your own corpus or replace an end-to-end evaluation of a retrieval product.
Technical notes+
pyproject.toml declares the console script mteb = mteb.cli:main (mteb/cli/__init__.py re-exports main from build_cli), requires-python >=3.10,<3.15, and many optional-dependency groups including api (FastAPI, uvicorn, OpenTelemetry), leaderboard (gradio==6.0.1), github (PyGithub) and per-model extras such as jina, nvembed and vllm. mteb/__init__.py exports evaluate, get_tasks, get_benchmark, get_model, ResultCache, mock_run and a deprecated MTEB evaluator class. mteb/models/__init__.py exposes EncoderProtocol, CrossEncoderProtocol, SearchProtocol plus wrappers for OpenAI-style APIs, sparse encoders, multi-vector models, compression and hybrid search. docs/contributing/adding_a_model.md requires model files to defer torch and transformers imports to call time so listing model metadata stays fast; docs/contributing/submitting_results.md documents ResultCache.submit_results with create_pr.
Observed
- License
- Apache-2.0 (LICENSE, pyproject.toml)
- Language
- Python, requires >=3.10,<3.15 (pyproject.toml)
- Install
- pip install mteb or uv add mteb (README.md)
- Interfaces
- Python library and an mteb CLI with run, available-tasks, available-benchmarks, create-meta, leaderboard and mock-run commands (docs/get_started/usage/cli.md)
- Declared maturity
- Development Status :: 5 - Production/Stable classifier (pyproject.toml)
- Model protocols
- Encoder, CrossEncoder and Search protocols for custom models (mteb/models/__init__.py)
- Results submission
- Results go to a separate results repository via pull request, then appear on the leaderboard (docs/contributing/adding_a_model.md)
- Optional extras
- Per-model dependency groups plus image, audio, video, api and leaderboard extras (pyproject.toml)
Read from README.md, pyproject.toml, LICENSE, mteb/__init__.py, docs/get_started/usage/cli.md, docs/api/evaluation.md, docs/contributing/adding_a_model.md, docs/contributing/submitting_results.md, mteb/benchmarks/__init__.py, mteb/models/__init__.py, docs/overview/index.md, mteb/cli/__init__.py.
Intel on MTEB
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.