UAE researchers develop first benchmark for AI across 13 Arab dialects

Abu Dhabi-based Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) researchers have unveiled a groundbreaking benchmark to evaluate AI models' capacity to comprehend and engage with Arab culture across 13 distinct dialects. The study, led by Professor Fajri Koto and Muhammad Dehan, highlights a significant disconnect between AI's ability to comprehend spoken Arabic and its proficiency in conversational usage.

Arabic unites over 400 million speakers globally, yet current AI models primarily trained on Modern Standard Arabic often lack the nuanced understanding required for everyday interactions. To rectify this, MBZUAI developed ArabCulture-Dialogue, a benchmark encompassing multi-turn conversations across Modern Standard Arabic and 13 national dialects from across the Arab world.

The researchers engaged 26 native Arabic speakers from 13 countries to create authentic dialogues covering 12 everyday topics, including weddings, food, parenting, agriculture, arts, and games. While models excelled at recognizing culturally appropriate responses (scoring in the mid-90s), their performance plummeted when asked to generate dialect or continue a conversation in that dialect.

The study revealed that models performed better with universally applicable customs, while country-specific cultural nuances proved challenging. North African dialect dialogues and Emirati dialogues were particularly difficult for the models, achieving dialect generation correctly in only about half of the cases.

Professor Fajri Koto emphasized that the models possess cultural knowledge but require guidance to apply it effectively. Specifying the country and region associated with a conversation improved accuracy significantly. The findings underscore the importance of developing AI systems that not only understand Arabic but also contextualize it within the diverse cultural landscape of the Arab world, especially for the UAE, which has placed AI development at the forefront of its national strategy.