48-Hour Field Notes: I Printed the Preferred Encoding. That Was Not the Open Default.

The writer spent an evening convinced their patch was complete since the sample opened and the assertion printed the expected name. However, a traceback appeared when a clean job started. The bytes were valid UTF-8, yet the clean job mentioned an ASCII codec. The writer examined the diff, ran pytest, and reran the test, but the issue persisted.

After checking the sample bytes and confirming they were UTF-8 without a BOM, the writer ran the pytest in the editor terminal and a fresh local shell, both times with green results. They then copied the job command onto their laptop without the job environment, which masked the real difference for another hour. The writer then requested another rewrite, submitting only the failing assertion, which received a new expected string that didn't mention locale.

The writer realized they had pasted the wrong world and needed to include the locale and interpreter flags in their paste. The writer's understanding deepened when they ran the test in a shell closer to a fresh image, exporting LANG and LC_ALL as C. After printing every encoding helper the standard library was willing to show, the writer discovered that Python text mode doesn't mean UTF-8 on every machine.

The implicit open consults a potentially ASCII-compatible encoding, while a non-ASCII name raises an exception instead of returning a string. A suggested solution was to trust the preferred encoding, but the writer emphasized the importance of checking which helper open actually consults before blindly following advice. The writer concluded that PEP 540 describes UTF-8 mode, but it doesn't guarantee a fix for Path.read_text on every release.

The writer highlighted the importance of using locale.getpreferredencoding(False) and locale.getencoding() for a more accurate understanding of the locale encoding. The writer cautioned against relying solely on the print statement and emphasized the need to review the open() and locale documentation for the specific interpreter version.