Rare Failure Eval

About

Rare Failure Eval is a reproducible Python experiment on agent evaluation under fixed budgets, comparing uniform and variance-based allocation of evaluation draws across 1,024 simulated worlds. It argues that an evaluator can make more wrong picks yet lose less utility, because selection accuracy differs from the cost of a mistake when failures are rare.