How 'Shōshō' Became 'Shomo' — Permission Character List Was Trimming Japanese

The original Japanese query for the inquiry desk, "しょうしょうおまちください," was turned into "しょもおまちください" after going through the text-to-speech (TTS) process. The discrepancy occurred because the model input was corrupted, which led to a missing character. The problem lay within the preprocessing stage of the TTS pipeline, where certain Japanese characters were not being recognized or preserved correctly.

The preprocessor had a whitelist that removed certain symbols, emoticons, and special characters. However, this whitelist did not include the character "々," which is classified as a symbol rather than a kanji character in Unicode. This oversight caused "しょうしょうおまちください" to become "しょもおまちください," with the "々" character missing.

Upon investigation, it was discovered that there were seven other missing characters, including "〆," "〇," "々," "㐀-䶿," "豈-﫿," and compatibility ideographs. These missing characters caused issues in the final synthesized audio, leading to mispronunciations and incorrect meanings. For example, "10〜20分" became "1020分," changing the meaning of the sentence. Additionally, the pronunciation dictionary was not applied, resulting in further inaccuracies in the output.

To resolve the issue, the solution was to convert the input text to Japanese before preprocessing. This involved creating a more comprehensive list of allowed characters, including those necessary for Japanese reading that fall outside the CJK Unified Ideographs block, such as "々," "〆," and "〇." By making this adjustment, the TTS process was able to correctly synthesize the intended audio, producing accurate pronunciations and meanings in the final output.