AI Models Swayed by 'Conversational Pressure' and Misinformation, Study Finds
University of Arizona researchers warn that LLMs can be manipulated by argumentative prompts, raising safety concerns for high-stakes deployment.
Generative AI models are susceptible to "conversational misinformed pressure," often accepting false statements when pushed during multi-turn interactions. A study from the University of Arizona warns that this inconsistency makes blind reliance on AI dangerous in critical decision-making environments.
Researchers tested seven large language models (LLMs): ChatGPT (GPT-3.5, GPT-4o, GPT-4o-mini), Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama-3-70B, and DeepSeek-R1. Published in Nature's Scientific Reports, the findings reveal that models frequently reaffirm misinformation when faced with repeated false statements. ChatGPT 3.5 was identified as the most vulnerable to this pressure, while Claude 3.5 Sonnet proved the most resistant.
The study also highlighted a specific behavioral pathology termed "reverberation," where models oscillate between accepting and rejecting the same false statement. While some models—specifically ChatGPT 4o, ChatGPT 4o-mini, Gemini 1.5 Pro, and DeepSeek—corrected errors 100% of the time when given a second opportunity, others remained unstable. DeepSeek was noted as the most "persuadable" when faced with argumentative prompts, a trait the researchers attributed in part to sarcastic responses that were difficult to interpret.
The Role of Training Data
Unlike one-off interactions, this research focused on multi-turn conversations to mirror real-world usage. The team found that all tested models were more susceptible to misinformation on obscure topics. This suggests a direct correlation between the volume of training data and a model's ability to resist false claims; the less a model "knows" about a topic, the easier it is to mislead.
Implications for High-Stakes AI
Led by Dr. Marvin Slepian, a Regents Professor of medicine and biomedical engineering at the Arizona Center for Accelerated Biomedical Innovation (ACABI), the study argues that these "fickle" systems lack the reproducibility required for high-stakes fields. In medical or military contexts, an AI's tendency to change its answer based on the user's argumentative tone could lead to catastrophic outcomes.
"This underscores the need for careful human engagement and the danger of blind reliance," Dr. Slepian stated. He further questioned the viability of using systems that remain inconsistent over time, noting that these characteristics persisted even across a three-year study period.
The Path Forward
Addressing these systemic failures is complicated by the opaque nature of closed-source models like GPT and Claude, which prevent researchers from diagnosing the root cause of the behavior. As AI integration expands, the study suggests that the industry must move beyond treating hallucinations as random errors and instead address the inherent pathologies of conversational pressure to ensure safety and reliability.