{
  "id": 4988270,
  "title": "How 'Shōshō' Became 'Shomo' — Permission Character List Was Trimming Japanese",
  "url": "https://urgent.news/2026/09/02/how-shosho-became-shomo-permission-character-list-was-trimming",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-02T01:24:17.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/orca_forge/how-shosho-became-shomo-permission-character-list-was-trimming-japanese-10bi"
  },
  "original_language": "en",
  "account": "The original Japanese query for the inquiry desk, \"しょうしょうおまちください,\" was turned into \"しょもおまちください\" after going through the text-to-speech (TTS) process. The discrepancy occurred because the model input was corrupted, which led to a missing character. The problem lay within the preprocessing stage of the TTS pipeline, where certain Japanese characters were not being recognized or preserved correctly.\n\nThe preprocessor had a whitelist that removed certain symbols, emoticons, and special characters. However, this whitelist did not include the character \"々,\" which is classified as a symbol rather than a kanji character in Unicode. This oversight caused \"しょうしょうおまちください\" to become \"しょもおまちください,\" with the \"々\" character missing.\n\nUpon investigation, it was discovered that there were seven other missing characters, including \"〆,\" \"〇,\" \"々,\" \"㐀-䶿,\" \"豈-﫿,\" and compatibility ideographs. These missing characters caused issues in the final synthesized audio, leading to mispronunciations and incorrect meanings. For example, \"10〜20分\" became \"1020分,\" changing the meaning of the sentence. Additionally, the pronunciation dictionary was not applied, resulting in further inaccuracies in the output.\n\nTo resolve the issue, the solution was to convert the input text to Japanese before preprocessing. This involved creating a more comprehensive list of allowed characters, including those necessary for Japanese reading that fall outside the CJK Unified Ideographs block, such as \"々,\" \"〆,\" and \"〇.\" By making this adjustment, the TTS process was able to correctly synthesize the intended audio, producing accurate pronunciations and meanings in the final output.",
  "summary": "📝 Originally published (in Japanese) at forge.workstyle.tech . I received a report about the avatar for the inquiry desk: \"しょうしょうおまちください\" becomes \"しょもおまちください\". Since I had just retrained the voice model multiple times, I first suspected the model. To cut to the chase, the model, parameters, and cache were all normal , but the input text passed to TTS was corrupted . Debugging from the downstream…",
  "key_points": [
    "Original Japanese query corrupted during TTS process",
    "Missing character 々 caused \"しょうしょう\" to become \"しょも\"",
    "Seven other missing characters affected audio synthesis"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}