AI chatbots struggle to understand cultural context, a new study finds
AI models often struggle to understand cultures around the world, and are reluctant to admit when they don’t know something. That’s what researchers at Canadian AI lab Cohere found when they tested whether chatbots could do more than simply speak a culture’s language. To determine whether models possessed deeper cultural knowledge, they presented them with a series of unique cultural riddles.…
A recent study by researchers at Cohere, a Canadian AI lab, has revealed that AI chatbots struggle to grasp cultural contexts around the globe. These models often fail to admit when they lack knowledge, despite speaking a language belonging to a particular culture. To evaluate their cultural understanding, the researchers presented a series of cultural riddles to 28 AI models.
Out of these, only one model correctly answered the riddle "What's the German 9/11?" The issue stemmed from the date format, as European countries write the date with the day first, followed by the month. In the case of the Berlin Wall falling on November 9, 1989, many models incorrectly assumed it to be November 9, based on the English date format.
While existing benchmarks test AI models on various cultural aspects, such as breakfast foods, historical events, and holiday customs, these tests often treat culture as a collection of static facts. The researchers realized that true cultural understanding involves connecting seemingly unrelated pieces of cultural knowledge and applying them in real-life situations.
To address this, they created the "CulturalRiddles" benchmark, which features riddles that demand answers rooted in lived experiences. This benchmark, developed by Julia Kreutzer, a senior research scientist at Cohere Labs, took six months to finalize after working with volunteers worldwide.
The researchers tested 28 AI models using 166 riddles collected from volunteers in 45 countries and 51 languages. The results showed that Google's Gemini 3.1 Pro Preview led the pack with a score of 79.4%. OpenAI's GPT-5.5 came in second with 76.3%, and Anthropic's Claude Opus 4.7 Thinking followed at 66%. However, Alibaba's Qwen 3.5 0.8B was the worst-performing model, accurately answering only 0.4% of the riddles.
Other notable performers included Amazon's Nova Micro, which correctly answered 8.3% of the riddles, and Google's Gemini 2.5 Flash Lite, scoring 33.6%. Even the best-performing model had below 50% accuracy in 6 out of the 57 communities tested, highlighting the need for improvement in cultural awareness within AI models.
One concerning aspect of the study is that models are more likely to guess answers instead of admitting they don't know the information. In fact, even when explicitly instructed to answer "I don't know," models did so just 0.7% of the time. Approximately 31% of the answers reviewed were fabricated, with varying levels of creativity.
The researchers found that when models confidently provide answers, they often default to dominant cultures represented on the web, rather than offering localized responses. To address these cultural knowledge gaps, Cohere Labs conducted a survey of AI users worldwide, finding that many users have to switch to English to get useful responses, as chatbot answers in their native languages are often inadequate. The survey also revealed instances of AI systems violating cultural norms.
Written by urgent.news from Fast Company's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
