Cloudflare saves 100 TB of RAM again, this time by slashing server hashes by 90% — cutting 100,000 entries down to 10,000 eliminates massive cache bloat

Cloudflare has successfully saved 100 terabytes (TB) of RAM by refining its hash-mapping algorithm. The company's primary use case for caching involves serving URLs directly from memory or disk, bypassing the need to fetch data from the live site, which can be time-consuming. To achieve this, Cloudflare employs its open-source Pingora framework, utilizing the Ketama algorithm for mapping URLs to cache servers. This process is intricate due to the dynamic nature of backend servers and the scale at which Cloudflare operates.

Initially, URLs are hashed using a method akin to calculating the cyclic redundancy check (CRC-32) for files, and then these hashes are matched to servers based on proximity in numerical terms. While this approach works, it results in an uneven distribution, causing overloading of servers when one fails. To address this, Cloudflare introduced additional hashes per server and randomized their arrangement, ensuring more equitable request distribution even after server failures.

However, this led to an excessive amount of memory usage, with Cloudflare managing up to 100,000 server hashes per machine. The team recognized that 100,000 hashes were far beyond the threshold of diminishing returns and conducted calculations to determine that a mere 10% of this amount would suffice for nearly identical results.

Employing optimizations within Rust's data structures, they managed to reduce the memory footprint by 2 bytes per entry, significantly impacting the overall reduction. Consequently, Cloudflare achieved a total savings of 100 TB of RAM through this optimization. They have also implemented this new algorithm as a separate code path, allowing for easy reversion if needed, while emphasizing the importance of resource-conscious software development.