Clip transcript
prompt. They should be retrieved when needed. So, moving to the next slide, let's talk about how to evaluate this fairly. We measure four things here: tool selection accuracy, time to first token, input tokens per request, and the estimated cost per thousand call. So, here for data sets, we used Berkeley function calling leaderboard, Skills bench style scenarios, and synthetic tool pools that allow us to scale the number of tools, so we can test at 10, 50, 100, 200, or even 1041 tools. So, we ran the same queries in two modes, the fat agent one and the semantic routing. Same model, same answer key, same tool catalog. The only difference was whether the model saw every tool or only the routed tool. We also swept K at 3, 5, 10 to understand the trade-off. So, the smaller K is the faster and cheaper. Larger K can be the can recover more edge cases. In practice, K equals 5 is a strong default starting point. So, if you are want to try this, you can default it to five. Here, K means the number of tools the semantic router retrieves and gives to the LLM for each user queries.