Reranker Simple Benchmark
github.com/instructkr/reranker-simple-benchmark- Category
- AI Tools
- Rank
- No. 2116Tools index
Previous survey · No. 1977 ·
- Listed in
- #48 Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Builder
- instructkr
- GitHub
- 36 stars
- Date
About
Make running benchmarks simple yet maintainable — Korean cross-encoder reranker evaluation, designed for repeatable experiments.
What it does
It builds a two-stage Korean search evaluation. BM25 first narrows each query’s document pool, then cross-encoder models rescore the leading candidates. It calculates standard retrieval metrics and presents saved results through interactive tables.
Why it's ranked here
This is a focused, credible benchmark harness with broad Korean coverage. It evaluates 18,945 queries across ten datasets, supports several reranker families, and separates candidate retrieval from model comparison. Its narrow scope and demanding GPU assumptions keep the verdict measured.
What's good
The benchmark covers Wikipedia, health, finance, law, commerce, long documents, and web FAQs. Model adapters normalize different scoring interfaces. Reports compare reciprocal rank, average precision, and discounted gain at several cutoffs, while separate tables expose per-dataset and averaged results.
Tradeoffs
The reranker runner assumes CUDA devices and uses reduced-precision model loading, so CPU-only evaluation is not its intended path. Some model and dataset loaders permit repository-supplied remote code. Korean tokenization also depends on external language packages, and MeCab requires a configured dictionary location.
How to use it well
Use it when comparing cross-encoder rerankers for Korean retrieval research, especially with multiple GPUs and reusable first-stage candidates. Keep dataset revisions, candidate files, model settings, batch size, and cutoff fixed between runs. It does not supply model training, production serving, or a general search application.
Technical notes+
pyproject.toml defines a Python 3.12 project with pinned or bounded dependencies including bm25s, MTEB, Sentence Transformers, Streamlit, and Torch. eval/retrieve_stage1_bm25.py implements BM25 retrieval with Mecab, Kiwi, Okt, and Kkma tokenizers. eval/evaluate_reranker.py distributes model-task jobs across CUDA GPUs, loads prior JSONL candidates, patches two MTEB dataset loaders, and selects wrappers by model-name patterns. eval/wrappers.py adapts Qwen3, Mxbai, BGE Gemma, and Jina scoring behavior. eval/custom_mteb_tasks.py maps two Hugging Face datasets into MTEB retrieval structures. eval/leaderboard_bm25.py and eval/leaderboard_reranker.py read saved JSON results into Streamlit tables.
Observed
- Primary language
- Python
- Runtime
- Python 3.12 or newer
- Packaging
- PEP 621 project metadata and dependencies are declared in pyproject.toml; README commands use uv
- Interfaces
- Command-line benchmark scripts and Streamlit result dashboards
- Platform requirements
- The reranker evaluation runner targets numbered CUDA GPUs
- Benchmark structure
- Two stages: BM25 candidate retrieval followed by cross-encoder reranking
- Project license
- No project license is stated in the supplied repository text
Read from README.md, pyproject.toml, eval/wrappers.py, eval/leaderboard_bm25.py, eval/custom_mteb_tasks.py, eval/evaluate_reranker.py, eval/leaderboard_reranker.py, eval/retrieve_stage1_bm25.py, .python-version, eval/results/stage1/bm25_Okt/model_meta.json, eval/results/stage1/bm25_Kiwi/model_meta.json, eval/results/stage1/bm25_Kkma/model_meta.json, eval/results/stage1/bm25_Mecab/model_meta.json, eval/results/stage1/bm25_Okt/Ko-StrategyQA.json.
What it can do
Run Korean cross-encoder reranker benchmarks
Korean text datasets and reranker models → Benchmark evaluation results
Evaluate reranker model performance
Trained reranker models and test datasets → Performance metrics and scores
Execute repeatable experiments
Experiment configurations and parameters → Reproducible benchmark results
Compare multiple reranker models
Multiple Korean reranker models → Comparative performance analysis
Generate benchmark reports
Evaluation results and metrics → Formatted benchmark reports
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.