A model can start answering quickly and still take a long time to finish. OpenRouter's provider guide separates time to first output from the rate at which subsequent text arrives. It also explains why median latency can conceal occasional long waits: a p99 measurement marks the threshold below which 99% of measured requests fall. Comparing these measures on an application's own workload gives developers a clearer view of responsiveness than a single headline speed figure. Provider evaluation guide
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
