Don’t stop early: Case-folding source code at memory speed

The issue of case-folding source code at memory speed is a significant concern for developers working with large amounts of text data. Case-folding is the process of converting text to a uniform case, typically lowercase, to enable case-insensitive comparisons. This operation is essential for various applications such as search engines, regex flags, and case-insensitive usernames and hostnames. GitHub, for instance, runs case folding on over 180 million repositories, amounting to more than 480TB of source code.

The key challenge lies in the speed of this operation, especially at such a massive scale. GitHub's Blackbird code search engine, for example, must perform case folding on every potential query result, making the efficiency of this operation crucial. The source material points out that the biggest performance improvement in the ASCII fast path came from removing an optimization rather than adding one.

Specifically, the authors found that it was faster to sweep the entire buffer without stopping early at the first non-ASCII byte rather than implementing a conditional break.

The authors emphasize that case folding and lowercasing are two distinct operations with different goals. Lowercasing is primarily for display purposes and is locale-sensitive, meaning it can vary based on the language and cultural context. In contrast, case folding is designed for comparison and is context-free, meaning it remains consistent regardless of the locale. The Unicode Character Database provides a CaseFolding.txt file to standardize this process.

The authors' crate, casefold, implements a simple and efficient case folding operation for ASCII characters only, which is the most common scenario. They deliberately exclude multi-character folds and Turkic locale-specific folds, as these are less common and could potentially introduce inconsistencies. They note that while this approach is common among other common tools and regex engines, maintaining consistency across tools is essential.

The authors then delve into the counterintuitive core of their approach: not stopping early in the folding process. Traditionally, developers might break the loop as soon as they encounter a non-ASCII byte, focusing on the cheap work for ASCII characters and deferring more complex operations to handle non-ASCII characters. However, this approach can be detrimental to performance, particularly on systems with high memory bandwidth.

The authors demonstrate that removing all branches and implementing a branchless loop results in significant performance gains, achieving speeds up to >45 GiB/s, which is essentially limited by memory bandwidth.

To achieve this level of optimization, the authors transform the operation into a branchless loop where each byte is processed uniformly, without conditional branches. They accomplish this by using bitwise operations to set a specific bit in a mask based on whether the byte is an uppercase ASCII character. This approach allows the loop to be fully vectorized, taking full advantage of the CPU's capabilities and achieving near-memory bandwidth speeds.

The final step involves ensuring that the early-exit condition is correctly handled, so that only the necessary parts of the buffer are processed. By keeping the break statement while making the body branchless, the authors maintain a balance between performance and functionality. This approach allows the loop to be fully vectorized, leading to a significant speedup compared to the more traditional branchy loop.

In conclusion, the casefold crate by GitHub demonstrates that by focusing on optimizing the most common case (ASCII characters) and employing branchless techniques, it is possible to achieve remarkable performance improvements in case folding operations. This approach not only enhances efficiency but also allows developers to handle large-scale text data more effectively, ultimately improving the performance of applications that rely on text matching and comparison.