A practical AI Evaluation pattern

Choosing an AI system by its best run can hide how often it fails on repeat attempts. In its evaluation guide, deepsense.ai describes testing ten analytical tasks five times per model. Adjusting its scores for variation between runs reversed the ordering of the two leading models.

The adjustment was the company's own scoring formula, and the two models' average scores were close (0.50 and 0.48), so the reversal is an illustration from one benchmark rather than a general ranking. It makes the issue concrete: average quality and repeatability can favor different systems. For work that must succeed repeatedly, evaluating the same tasks several times reveals a property that a single successful demonstration cannot establish.