TabBench-Bio
tabbench-bio.eu- Category
- Developer Tools
- Rank
- No. 1959Tools index
Previous survey · No. 1858 ·
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- Web
- GitHub
- 1 stars
- Latest release
- v1.0.0
- Date
About
A continuously updated benchmark comparing classical models, neural networks, and tabular foundation models on high-dimensional, low-sample-size biomedical tables (transcriptomic, metagenomic, molecular, sequence-derived, genomic-prediction), using a controlled feature-by-sample grid and 5x cross-validation. It reports interactive leaderboards, Bradley-Terry Elo rankings, efficiency and stability analyses, and an explicit policy for scoring out-of-memory failures rather than substituting smaller runs.
What it does
A Python package and command-line runner for biology datasets that carry far more measurements than samples, such as gene expression and methylation. It downloads the data, fixes the train and test splits, fits a roster of models on each, and ranks them by head-to-head wins per fold. You can drop in your own scikit-learn estimator and see where it lands against Random Forest and a constant baseline, or load the published results database and rank it without training anything.
Why it's ranked here
The scoring rules are written down and tested, which many benchmarks skip. A model that crashes on a dataset is scored at the constant baseline there, so failing where it struggles cannot lift its rank, while a model kept off a dataset by a size limit is left out rather than counted as a loss. Every run config must declare every setting, and an unknown or misspelled key stops the run. An arXiv paper backs the method and the code ships under the EU public licence. The package still calls itself alpha.
What's good
Fairness choices are explicit. Every single model runs one library-default setup with no tuning and no bagging, and the AutoGluon ensemble, which gets the heavy preset, is reported as a reference line instead of a ranked peer; a test fails if any other model opts out. Feature caps pick a random column subset using training rows only. Ranking intervals come from 2,000 bootstrap rounds that resample whole targets. Merging two result databases checks cell settings and held-out targets, drops duplicate attempts and refuses to overwrite an existing file.
Tradeoffs
Setup is heavy. The core pins exact numpy and scikit-learn versions and supports only Python 3.11 and 3.12. Registered models need a separate environment that installs a forked AutoGluon, because the stock version caps foundation models at 500 features, and one model installs straight from a Git repository. Some dataset sources need credentials, Kaggle among them. The quick-start helper has no fit timeout, cannot resume, scores only hard predictions with macro-F1 or RMSE, and fits its own Random Forest, so its numbers differ from the published ones.
How to use it well
Built for researchers with a new model for wide, small-sample biology data who want a fair comparison against published baselines. Start with the scikit-learn template on the two public gene-expression datasets, then rerun against the published results database on its frozen splits so only your model is fitted. Report that as a subset comparison, as the docs ask. Move to a registered adapter when you need resumable cluster runs. It is not a general tabular benchmark: every dataset is biological, with many features and few samples.
Technical notes+
src/tabbench_bio/config.py: load_config asserts that a config holds every key in REQUIRED_KEYS and nothing else except model_tuning; model_overrides flags any roster entry declaring presets, ensemble, optimize or num_hpo_trials, and config_for_cell pins medium_quality presets, no HPO, no ensembling and a median NaN policy for KNN. src/tabbench_bio/benchmark.py keys the processed split cache on a SHA-256 of every split parameter including _PREPARED_DATA_VERSION, asserts on metadata mismatch, and drops features missing in over half the training rows. tests/test_scientific_integrity.py covers grouped-fold leakage, failure imputation at the DUMMY metric, design-skip exclusion and the AUTOGLUON-only override. configs/models/all.json records advisory memory priors for MITRA, TABPFN-WIDE and TABFM, none enforced. .github/workflows/ci.yml runs ruff and pytest on 3.11 and 3.12.
Observed
- License
- EUPL-1.2, with a vendored Apache-2.0 Elo helper
- Language
- Python 3.11 and 3.12
- Interfaces
- Command-line tool and Python library
- Install surface
- Editable install from a Git clone with uv
- Declared maturity
- Alpha
- Ranking method
- Fold-level Bradley-Terry Elo with Random Forest fixed at 1000
- Tests and CI
- Pytest suite, ruff lint and format checks on Python 3.11 and 3.12
- Agent integration
- Markdown model-selection skill for agents
Read from README.md, pyproject.toml, LICENSE, src/tabbench_bio/cli.py, src/tabbench_bio/benchmark.py, src/tabbench_bio/config.py, benchmark_my_model/README.md, benchmark_my_model/my_model.py, configs/models/all.json, tests/test_scientific_integrity.py, .github/workflows/ci.yml.
What it can do
Benchmark classical models, neural networks, and tabular foundation models on high-dimensional, low-sample-size biomedical datasets
Biomedical tabular datasets (transcriptomic, metagenomic, molecular, sequence-derived, genomic-prediction) → Comparative model performance results
Run 5x cross-validation evaluation on a controlled feature-by-sample grid
Tabular dataset and model → Cross-validated performance metrics
Generate interactive leaderboards of model performance
Benchmark results → Interactive leaderboard
Compute Bradley-Terry Elo rankings across models
Model comparison results → Elo ranking scores
Produce efficiency and stability analyses of models
Benchmark run data → Efficiency and stability reports
Score out-of-memory failures according to a defined failure-handling policy instead of substituting smaller runs
Model run outcome → Recorded failure score
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.