Intel / article

Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses

Source: arxiv.org
Author: Mahsa Amani, Seungeon Lee, Abhisek Dash, Asmaa El Fraihi, Yunah Jang, Elisabeth Kirsten, Qinyuan Wu, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander, Muhammad Bilal Zafar, Soumi Das
Date: Published Sep 18, 2026

Context

An AI assistant searching the web more often does not necessarily produce better answers. A study of four conversational platforms combined real user interactions with controlled API experiments, examining when systems searched, how they phrased queries and how retrieved material became answers. It found substantial differences in search behavior without a simple relationship between search frequency and response quality. Study abstract.

The study also reports that some answer claims depended on retrieved pages that were not cited. That makes source retrieval and visible attribution separate parts of the problem: an answer can draw on search results while leaving readers unable to trace a particular claim to its supporting page. Attribution findings.

Key takeaways · AI-distilled

Comparing ChatGPT, Claude, Grok, and DeepSeek, the study found that invoking web search more often does not reliably produce better-quality responses.
Each platform's search engine tends to surface results from its own preferred domains, meaning the same query can return meaningfully different sources depending on which assistant runs it.
The four agents use noticeably different query-formulation strategies when deciding how to search, not just when to search.
Although responses are mostly in retrieved search results, the study found some claims rely on uncited search results, raising attribution and reliability concerns.

Terms in this piece · Glossary

grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.

Read the source arxiv.org