Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses
Source
Mahsa Amani, Seungeon Lee, Abhisek Dash, Asmaa El Fraihi, Yunah Jang, Elisabeth Kirsten, Qinyuan Wu, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander, Muhammad Bilal Zafar, Soumi Das
Author
Mahsa Amani, Seungeon Lee, Abhisek Dash, Asmaa El Fraihi, Yunah Jang, Elisabeth Kirsten, Qinyuan Wu, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander, Muhammad Bilal Zafar, Soumi Das
Date
Key takeaways · AI-distilled
Comparing ChatGPT, Claude, Grok, and DeepSeek, the study found that invoking web search more often does not reliably produce better-quality responses.
Each platform's search engine tends to surface results from its own preferred domains, meaning the same query can return meaningfully different sources depending on which assistant runs it.
The four agents use noticeably different query-formulation strategies when deciding how to search, not just when to search.
Although responses are mostly groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → in retrieved search results, the study found some claims rely on uncited search results, raising attribution and reliability concerns.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
Search behavior varies widely across platforms, and searching more often doesn't reliably improve answer quality, a useful data point for tuning when to enable web search in an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →.
articleSimpleWikiSearch: A Clean Offline Wikipedia Environment for Agentic SearchGuanming Xiong, Penghui Zhang
articleBenchmarking Hybrid Deep Research Across Database Querying and Web SearchRuofan Wu, Peiran Xu, Xiaolong Li, Fan Shu, Soyoung Yoon, Yite Wang, Xiaodong Yu, Boyi Liu, Feng Yan, Debiao Li, Yuxiong He, Zhewei Yao
articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang