Vibeleaderboard
← All Intel
Intel / article

Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

Source
arxiv.org
Author
Heming Fu, Shan Lin, Qianqian Xie, Guojun Xiong
Date

Context

A cheap model call can become an expensive workflow if it repeatedly fails and retries. InflationAgent selects models using estimated accuracy relative to predicted total workflow cost. The authors report costs reaching 4.25 times the single-call cost for a seven-billion-parameter model on questions requiring multiple reasoning steps.

The router estimates difficulty before execution and accounts for expected retry overhead when choosing a model. When it escalates to a stronger model, it discards the failed reasoning chain. The proposal makes successful completion, including attempts that did not work, the relevant unit for comparing model costs.

Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Recommended reads
Comments

Checking sign-in…

Loading comments…