← Glossary
Glossary · AI and agent workflows
Latency
The time between asking for a result and having it, including the model's generation time, tool calls, retries and any queueing.
Latency matters differently for different tasks: a review that runs overnight can be slow; one needed during a meeting cannot. It is elapsed time, during which a person may do other work, and is not the same as their labor. Failed attempts add their elapsed time to the total. Local inference on a small machine can have high latency and no provider charge at once, which is why the two are separate columns.