Persuaded, Not Informed: Incentive-Misaligned Witnesses Defeat In-Context Grounding
Source
arxiv.org
Author
Rahul Balakavi
Date
Context
A sales conversation can contain confident assurances that conflict with a company's actual requirements. In a study of lead qualification, a model reading only call transcripts approved 29 of 31 deals where a representative's assurances contradicted the supplied price list or installation policy.
The researchers distinguish this from simply lacking information: adding records did not reliably make the model apply them correctly. Their experiments isolate the step that calculates whether budget and timing satisfy policy. The supported lesson is that an assertion in a business record and compliance with an explicit rule need separate treatment; the paper's negative generalization test limits broader claims.
Key takeaways · AI-distilled
On 100 CRMArena-Pro lead-qualification tasks, the sales rep asserted an acceptable timeline in every call and an acceptable budget in 76. Where that claim contradicted the price list and installation policy (31 tasks), a transcript-only model cleared the deal 29 times.
Seven models from four providers were misled 87-97% of the time, and the author reports that scale and explicit reasoning conferred no resistance.
Only 3 of 35 genuine failures involved no assertion, which the author reads as persuasion rather than missing information. Giving the model the company records lowered strict accuracy from 41 to 18 while raising recall, because precision collapsed.
In a control that holds extraction fixed and varies only who computes Budget and Timeline, the margin ranged from 42 points on an inexpensive model to 2-5 points on models that already compute correctly; on the strongest models the arms fall within confidence intervals.
A pre-specified generalization test returned a negative result, and the author limits the finding to a precondition: the policy must be exactly specified in the inputs.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleBeyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM YieldingGuang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian
articleContext Poisoning as Extreme-Value Attention Interference in Long-Context Language ModelsMeysam Ghaffari, Nina Fatehi, Bhaskar Sen, Nasim Sabetpour, Carlos Morato
articleCaught in the Story: Narrative Captivity in Multi-turn LLMs ConversationYuhe Wu, Guangyu Wang, Yujie Chen, Jiatong Zhang, Yuran Chen, Yutong Zhang, Xiyin Cheng, Wenpeng Cao, Zhuang Liu, Guang Zhang