OpenRouter benchmarks web search tools for grounding agents
- Source
- x.com
- Date
Introducing Web Search Benchmarks 🌐 Rankings of search tools across different models and configurations to help you decide how to ground your agent: https://t.co/GkyEte2YO0
We chose four diverse benchmarks, ran each with four models across four search depths on Exa, Parallel, Perplexity, and the model's native engine. We ranked all the combinations by quality, cost, and speed.
Of all the factors we tested, increasing the search budget had the biggest impact on results. Going from a budget of 1 to 25 turns roughly doubles the scores on the BrowseComp benchmark.
Swapping models had greater effect on score than swapping engines. The average difference in score between frontier and cost-efficient models was ~15 points. Swapping engines while holding the model constant shifted scores by ~10 points on average.
Context
OpenRouter's search benchmarks show why an 's search allowance belongs in a cost-and-quality comparison. On BrowseComp, a difficult fact-finding test, the company reports that raising the allowance from one search turn to 25 roughly doubled scores while increasing cost per question by 2.5 to seven times. That figure comes from its table of three models using one search provider, and its benchmark report limits the setup to search-result excerpts, with full-page fetching and code execution disabled.
Additional searching also accumulated on unsuccessful attempts. With a 25-turn allowance, incorrect BrowseComp answers averaged 19.7 searches, compared with 10.3 for correct answers. This suggests evaluating search depth against the actual workload: a larger allowance can improve difficult research, but can also fund longer unsuccessful attempts. The reported breakdown makes both effects visible.
Checking sign-in…
Loading comments…