What Lowercasing Taught Me About Trusting Strings

A post reminds the reporter that the most dangerous line of code in a system is often the one that looks harmless. This case involves calling the .lower() function on a string, which can lead to a security vulnerability. The reporter initially dismissed this as skepticism, but came to understand the truth behind it. Lowercasing a string is not a simple, mechanical process; it's a linguistic operation governed by Unicode, and can have unexpected results.

Two examples illustrate this: Turkish has a dotless "i," and the lowercase conversion varies based on locale settings. Additionally, characters outside ASCII can have lowercase forms that are ASCII characters themselves, potentially changing a string's content after normalization. This can lead to a time-of-check versus time-of-use bug, where a system validates one value but acts on a different one after normalization.

The reporter emphasizes that this issue is not limited to specific languages or Unicode versions; it's a general problem in string transformation. The fix is not to eliminate the transformation, but to reorder the code: canonicalize the string once at the boundary, then validate the canonical form. This approach ensures that the value is settled and untouched downstream, similar to how HTML escaping and SQL parameterization are handled.

The reporter also questions the need for locale-awareness in security comparisons, suggesting that such decisions should prioritize boring, consistent rules over complex, locale-dependent behavior. Ultimately, the key takeaway from this experience is to identify and address the points in code where a value is validated and then transformed, as these are the areas most prone to security bugs.