UTF-8 vs. cp932: Why Some Bugs Only Show Up on Japanese Windows
"It works fine on my Mac, but it crashes the moment we hand it to Windows" is a report a lot of developers eventually hear. More often than not, the culprit is a character encoding mismatch. This post walks through the basics of text encoding, then looks at a real build-pipeline crash this project ran into, and the test written to make sure it never happens again. What an Encoding Actually Is…
When software doesn't behave as expected on Windows with Japanese settings, the issue often stems from character encoding problems. At its core, character encoding maps symbols to strings of bytes for storage or transfer. Different encodings can produce varying byte sequences for the same symbol. For example, the Japanese character "あ" turns into either b'\xe3\x81\x82' in UTF-8 or b'\x82\xa0' in cp932, which Windows uses for Japanese locales.
If a program encodes a string using one encoding and later decodes it using another, garbled text or errors appear. UTF-8 supports nearly all characters, including emojis, while cp932, specific to Japanese text, does not include most emojis. This discrepancy is where bugs commonly occur.
The source explains that some applications only crash on Windows due to encoding mismatches. Python, when capturing a subprocess's output via a pipe without explicit encoding, assumes the platform's default encoding (UTF-8 on macOS and Linux, cp932 on Windows). Thus, a string containing emojis works fine on macOS but crashes on Windows, reproducing only when the code runs on a Windows machine.
In a specific incident, a build script named build_app.py ran a version-checking script named bump_version.py. When this script printed an emoji, the subprocess captured its output using UTF-8 on macOS and cp932 on Windows, causing a UnicodeEncodeError and crashing the build process. To prevent such issues, tests were created to catch encoding misalignments before code reaches deployment.
These tests include a static check for characters that can't be encoded in cp932 and a behavioral check that simulates the Windows environment's encoding to ensure the script runs correctly under those conditions.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
