Persuaded, Not Informed: Incentive-Misaligned Witnesses Defeat In-Context Grounding
Source
Rahul Balakavi
Author
Rahul Balakavi
Date
Key takeaways · AI-distilled
On 100 CRMArena-Pro lead-qualification tasks, the sales rep asserted an acceptable timeline in every call and an acceptable budget in 76. Where that claim contradicted the price list and installation policy (31 tasks), a transcript-only model cleared the deal 29 times.
Seven models from four providers were misled 87-97% of the time, and the author reports that scale and explicit reasoning conferred no resistance.
Only 3 of 35 genuine failures involved no assertion, which the author reads as persuasion rather than missing information. Giving the model the company records lowered strict accuracy from 41 to 18 while raising recall, because precision collapsed.
In a control that holds extraction fixed and varies only who computes Budget and Timeline, the margin ranged from 42 points on an inexpensive model to 2-5 points on models that already compute correctly; on the strongest models the arms fall within confidence intervals.
A pre-specified generalization test returned a negative result, and the author limits the finding to a precondition: the policy must be exactly specified in the inputs.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
If your AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → reads business records containing assertions from parties with an incentive to mislead, scale and chain-of-thoughtHaving a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.Full definition → don't protect it; test for this groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → failure before deploying agents on CRM-style data.
articleContext Poisoning as Extreme-Value Attention Interference in Long-Context Language ModelsMeysam Ghaffari, Nina Fatehi, Bhaskar Sen, Nasim Sabetpour, Carlos Morato
articleCaught in the Story: Narrative Captivity in Multi-turn LLMs ConversationYuhe Wu, Guangyu Wang, Yujie Chen, Jiatong Zhang, Yuran Chen, Yutong Zhang, Xiyin Cheng, Wenpeng Cao, Zhuang Liu, Guang Zhang
articletrajectory-judge: What Outcome-Only LLM Judges Miss on Agent TrajectoriesHadi Mohammadi