illate-parity
github.com/illate/parity- Category
- Developer Tools
- Rank
- No. 3082Tools index
Previous survey · No. 2891 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- CLI
- Date
About
A Python command-line tool that checks whether a cheaper model performs as well as the one currently in use. It scores each model from prediction files with bootstrap confidence intervals, then runs a non-inferiority test to see whether the candidate stays within an agreed margin of the current model at 95% confidence. It needs no GPU or API keys and depends only on NumPy.
At a glance
illate-parity compares two classification models using prediction files matched by example ID. It is worth using before switching to a cheaper model if you have labeled examples and can say how much accuracy you are willing to trade.
Good fit
Teams evaluating a replacement classifier against the same held-out examples as the current model.
Consider something else when
Use a task-specific evaluation if your outputs need human judgment or if your primary metric cannot be reduced to the documented classification scores.
Start here
Choose an acceptable accuracy drop before looking at results. Compare both prediction files against the same gold labels and inspect the interval and per-class failures. Treat an inconclusive result as a reason to collect more examples.
Cost and requirements
The comparison consumes local JSONL files and uses NumPy. It needs no GPU or API key for scoring. Producing the two sets of predictions is separate work.
Limits to know
Without gold labels, it measures agreement with the current model, including its mistakes. Even a pass applies to the sample and margin you chose.
Sources checked 2026-11-05. Source-based assessment.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.