Vibeleaderboard
← All Intel
Intel / post

Etalon: measuring LLM streaming latency beyond TPOT averages

Source
x.com
Date
wafer_ai@wafer_ai

this is a must-read if you want to be a top 1% ai performance engineers follow and save to keep up with the series. links in thread 🧵 part 8: Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems Amey Agrawal and coauthors explain how serving decisions shape the timing of streamed LLM responses. for AI performance engineers evaluating batching, chunked prefill, or speculative decoding, the paper provides a framework for assessing token delivery against application-specific timing targets. the authors connect scheduling behavior, token arrival traces, and service targets: - prefill and decode interference: a long prefill can delay ongoing decode work. chunked prefill divides that prompt processing across batches, so the amount of prefill work per batch can affect both first-token latency and interruptions to existing streams. - first-token latency: time to first token (TTFT) includes scheduling delay and prompt processing. the authors propose profiling isolated requests across prompt lengths and adding a scheduling allowance to set first-token deadlines. - averages and latency distributions: time per output token (TPOT) can hide generation pauses,…

Read the full post on X

Context

A streamed AI response can post a good average speed and still stall partway through. The Etalon paper by Amey Agrawal and coauthors offers a way to measure those stalls. In a summary by the wafer_ai account, part 8 of its reading series, the usual metric, time per output , averages over the whole response and can hide pauses. Percentiles of the time between tokens show how long the gaps get, but not where in the response they fell. Etalon instead keeps a per-request trace of when each token reached the client.

Etalon judges that trace against deadlines set by the application's timing targets, such as the rate at which the client displays tokens. Its fluidity index is the fraction of deadlines met within a request. Tokens that arrive early build slack, which matters when a technique such as delivers several tokens in one burst that the client can buffer for steady display. When a token misses its deadline, Etalon counts the missed deadline slots and resets later deadlines from that token's arrival. The first token gets its own deadline, set by profiling isolated requests across prompt lengths and adding an allowance for scheduling delay.

From these measures Etalon derives a fluid token generation rate, the fastest playback rate that meets a chosen fluidity target for a specified share of requests, which the summary says supports comparing APIs. It also estimates the highest request rate one serving replica can sustain under the targets on a tested workload, which feeds estimates of how many replicas a deployment needs. The thread's text describes the method but does not report the paper's measured results.

Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
More from wafer_ai
Recommended reads
Comments

Checking sign-in…

Loading comments…