Are LLMs good or bad Bayesians?

Bryan Caplan asserts that artificial intelligences, including himself, exhibit poor Bayesian reasoning. When confronted with contradictory evidence, these AI systems should either adopt the opposing viewpoint or remain unchanged, depending on whether the argument was more or less favorable than anticipated. However, ChatGPT appears to either adjust its position or fail to change at all in response to such counterarguments.

Caplan identifies this as a significant flaw in AI reasoning, where LLMs tend to treat a plausible counterargument as evidence supporting their conclusion, rather than comparing it to their initial expectations.

While Caplan concedes that there is no general theorem mandating equal probability updates towards and away from a speaker, he argues that under suitable calibration assumptions, updates in both directions should occur. Furthermore, he notes that even if an argument is technically valid, if it's weaker than expected, it should prompt a negative update.

The full dialogue discusses whether AI systems should attempt to rectify this issue and whether it can be achieved through effective prompting techniques. The post raises critical questions about the nature of Bayesian reasoning and its applicability to AI systems.