โมเดลและชุดข้อมูลไทยบน Hugging Face ที่คนส่วนใหญ่ยังไม่รู้ว่ามีอยู่

A major gap in knowledge among Thai AI workers is the availability of Thai language models and datasets on Hugging Face, a platform for sharing machine learning resources. While many are familiar with popular Thai models like OpenThaiGPT, the platform hosts a host of other resources that remain largely unknown. This article identifies key models and datasets that are underutilized and could benefit Thai AI development.

One crucial resource is the OpenThaiGPT evaluation dataset, which provides a standardized benchmark for comparing Thai language models. This dataset, licensed under Apache 2.0, ensures that researchers and developers can use it to evaluate their own models on a level playing field. Similarly, the WangchanThaiInstruct Multi-turn Conversation Dataset offers a collection of multi-round Thai dialogues generated through a process called prompt engineering. This dataset, released under CC-BY-SA 4.0, is particularly valuable for training chatbots in Thai.

Despite their potential, these datasets come with caveats. The OpenThaiGPT evaluation dataset, for instance, may be limited in diversity due to its construction process based on an inherently biased model. Researchers should be cautious about directly fine-tuning models on such datasets as this could potentially perpetuate the same biases.

Additionally, licensing terms vary among datasets, with some using Apache 2.0 (permissive) and others CC-BY-SA 4.0 (copyleft). Developers must carefully review these licenses before using datasets commercially.

Another important point is that evaluation datasets should not be the sole metric for assessing model performance. While larger download numbers might suggest quality, they do not necessarily reflect the dataset's suitability for a particular application. Moreover, datasets created by Thai speakers tend to outperform those generated by non-native speakers, emphasizing the importance of using data that reflects the language as it is spoken in Thailand.

For those new to Thai AI, the article recommends starting with evaluation datasets to understand a model's limitations, then seeking out specialized datasets to address specific weaknesses. By doing so, they can avoid redundant efforts and make more informed decisions about which models and datasets to use. Ultimately, building a robust Thai AI ecosystem requires not just the development of sophisticated models, but also the careful curation and sharing of high-quality datasets tailored to the unique characteristics of the Thai language.