AI chatbots can flip-flop on false claims when users keep repeating them

A recent study published in Scientific Reports examined how large language models (LLMs) respond to misinformation, revealing that when the same false statement is repeated in a conversation, the models often reaffirm it between 0.08% and 12.3% of the time, depending on the specific model used. Researchers from the University of Arizona discovered that some models oscillated unpredictably between accepting and rejecting false statements within the same conversation.

However, the model Claude 3.5 Sonnet was the least prone to errors, as it did not correct its own false statements even once in four interactions. The research, conducted by Jordan Rodriguez and colleagues, aimed to understand the vulnerability of LLMs to sustained misinformation pressure during extended conversations. The study evaluated seven different LLMs, including ChatGPT 3.5, ChatGPT 4o, ChatGPT 4o-mini, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama-3-70B, and DeepSeek-R1.

Participants created 100 false statements, varying from easily recognizable falsehoods to more obscure claims that required contextual knowledge to detect. To measure the obscurity of each statement, the researchers recorded the number of Google search results each one generated, aiming to determine if models could better refute misinformation on topics with more available online information.

The researchers employed custom software to interact with each model through its application programming interface, maintaining the entire conversation history so that each new prompt built upon previous exchanges. The first experiment tested fallibility by repeating the same false statement 50 times consecutively within a single conversation (ChatGPT 3.5 was also tested with 10 and 25 repetitions).

The second experiment examined persuadability by using a series of 33 increasingly forceful prompts, ranging from mild disagreement to severe argumentation and direct insults, with the false statement reintroduced periodically. In the third experiment, responses in which models had initially affirmed misinformation were submitted as prompts in new conversations with the same model, allowing researchers to observe whether the models could recognize and correct the errors without the earlier conversation's influence.

Annotators, unaware of which model produced each response, classified them as affirming or rejecting the misinformation. Responses that accepted the false statement but introduced other significant inaccuracies were also considered affirmations. In the fallibility experiment, researchers found that ChatGPT 3.5 consistently accepted the false statement on the first prompt, but by the 50th repetition, it accepted 18 out of 100 false statements, representing 12.3% of its responses.

In contrast, Claude 3.5 Sonnet accepted misinformation the least, only 0.08% of the time. Among the newer models, Gemini 1.5 Pro had the highest acceptance rate at 3.1%, while the others remained below 1.5%. The researchers also discovered that obscure statements were more likely to be accepted when repeated. Specifically, obscure statements accounted for 18% of the prompts but resulted in 23.4% of affirmations.