Vibeleaderboard
Index / tool
Visit github.com
Category
Developer Tools
Rank
No. 3082Tools index

Previous survey · No. 2891 ·

Pricing
Open Source
Type
TOOL
Use case
Model & Agent Evaluation
Interfaces
CLI
Date

About

A Python command-line tool that checks whether a cheaper model performs as well as the one currently in use. It scores each model from prediction files with bootstrap confidence intervals, then runs a non-inferiority test to see whether the candidate stays within an agreed margin of the current model at 95% confidence. It needs no GPU or API keys and depends only on NumPy.

At a glance

illate-parity compares two classification models using prediction files matched by example ID. It is worth using before switching to a cheaper model if you have labeled examples and can say how much accuracy you are willing to trade.

Good fit

Teams evaluating a replacement classifier against the same held-out examples as the current model.

Consider something else when

Use a task-specific evaluation if your outputs need human judgment or if your primary metric cannot be reduced to the documented classification scores.

Start here

Choose an acceptable accuracy drop before looking at results. Compare both prediction files against the same gold labels and inspect the interval and per-class failures. Treat an inconclusive result as a reason to collect more examples.

Cost and requirements

The comparison consumes local JSONL files and uses NumPy. It needs no GPU or API key for scoring. Producing the two sets of predictions is separate work.

Limits to know

Without gold labels, it measures agreement with the current model, including its mistakes. Even a pass applies to the sample and margin you chose.

Tags

model-evaluationnon-inferioritybootstrappythonfine-tuningcli

Tech Stack

Python

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.