{
  "id": 7962958,
  "title": "We Asked 330 Models a Question in Korean. Half of Them Answered in the Wrong Alphabet.",
  "url": "https://urgent.news/2026/09/17/we-asked-330-models-a-question-in-korean-half-of-them-answered-in-the",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-17T05:54:10.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ai_maya_063fc568e157562fd/we-asked-330-models-a-question-in-korean-half-of-them-answered-in-the-wrong-alphabet-la2"
  },
  "original_language": "en",
  "account": "The article reports on testing 330 language models on Korean language skills across seven categories. Before any grading, a function removed answers deviating from Korean, catching 33% of the models that failed the test. Of the failing models, 52% had errors in multiple categories. The hardest tests were those requiring strict adherence to Korean script—honorifics, terminology, and institutions. The article argues that an LLM judge can grade a mixed-script answer as natural despite the language error. To filter out these errors, the authors implemented a script that flags answers with less than 25% Hangul characters, Hanja (Chinese characters), or Kana (Japanese characters), which automatically fails the answer. This filtration reduced the number of judge evaluations by 766 cases. The grading results showed only 7.6% of models achieved an A grade overall, with 33.6% receiving an F. The most promising axes were honorifics (8.5% A rate) and format compliance (47.6% A rate). Interestingly, newer models did not consistently outperform older ones on this specific metric. The authors recommend implementing the countable check as the first step in the evaluation process to filter out non-Korean answers before they reach the judge, potentially saving costs and improving grading accuracy.",
  "summary": "We graded 330 language models on Korean across seven axes. Before any of that, a four-line function threw out a third of the answers. That function turned out to be the most useful thing in the harness. The check def contamination ( text : str ) -> dict : total = len ( text . strip ()) if not total : return { \" hangul_ratio \" : 0.0 , \" hanja \" : 0 , \" kana \" : 0 , \" broken \" : True } h = len (…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}