← All IntelClip / AI Agents99% token reduction and the 5-second time-to-first-token wall
From The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica · ≈5:45
Puts cost and latency numbers on the trap: at 100k requests/day you ship billions of tokens just describing tools, and 500 tools can push first-token latency past 5 seconds.
What’s in it
- Puts cost and latency numbers on the trap: at 100k requests/day you ship billions of tokens just describing tools, and 500 tools can push first-token latency past 5 seconds.
Clip transcript
So, two more reasons over here. First is the latency and the second is the cost. So, we saw in the earlier slide that accuracy was a big problem. Another issue is with latency and cost over here as we can see. So, like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens. That will include the tool description and the schema text. So, that cost is basically being paid on every request. At 100,000 requests a day, if you push this into production, you are sending in billions of tokens and just to describe those tools. And with just-in-time routing, the prompt may include only three to five relevant schemas, closer to about 1,000 tokens. That is roughly a 99% reduction in tool context tokens. Another issue is latency. With fat agent approach, time to first token grows as the tool catalog grows because the model has to process a larger prompt before it can answer the user question. So, say for example, if you have 500 tools in your agent, the fat agent path can push first token latency past 5 seconds. So, this is specifically important if your product is real-time because this will take longer time to respond. So, which will make the design feel slow and unpredictable. So, and with semantic routing, it might feel more responsive and it will be more real-time in a way.
Comments
Sign in to comment.
Loading comments…