Urgent.News

What's breaking now, across hundreds of outlets.

Tech

One Hash Bloom Filters

Abstract editorial illustration

One Hash Bloom Filters: A Breakthrough in Efficiency

Probabilistic data structures known as bloom filters enable the swift and economical verification of whether a specific key, X, is present within a set. These filters strike a balance between memory consumption and computational efficiency, making them a popular choice in high-performance data systems. Utilized by Bluesky, these filters have notably accelerated query processing, cutting disk IO and enabling the handling of millions of queries per second.

Bloom filters consist of an array of bits and k hashing functions. A key is added to the set by executing it through the k hash functions, subsequently marking the corresponding bits in the array. During verification, the key is hashed k times, and the resulting positions are inspected. If any bits are found to be zero, the key was not incorporated into the set. Conversely, if all bits are set, it is probable that the key is within the set.

However, the traditional two-hash approach can be computationally expensive in high-performance scenarios. In 2008, Kirsch and Mitzenmacher introduced a technique demonstrating that two hashes could yield similar results. They utilized the formula (hash_1 + i*hash_2) % n to determine the i-th location, maintaining a comparable false positive rate with a reduced computational cost.

Bluesky adopted the double-hash bloom filter implementation from Daniel Lemire's repo, "bits-and-blooms/bloom," which runs two 128-bit murmur3 hashes to produce four 64-bit values, subsequently amalgamating them to generate all k bit locations. The Bluesky backend routinely loads data into these filters upon process initialization, with an additional real-time atproto firehose consumer populating the filters during process startup.

However, this implementation lacked thread-safety, necessitating mutex implementation to prevent data races.

Jasmine forked the library, creating "atomic-bloom," an implementation that harnesses atomic CPU instructions to execute the same logic without mutexes. This optimization led to a significant reduction in mutex contention, resulting in faster deploys from 90 minutes to approximately 15 minutes. While Amdahl's Law indicated no further speedup due to contention, the overall impact was still substantial.

In pursuit of further optimization, the Bluesky team explored a single-hash bloom filter technique. They developed "jcalabro/gloom," a one-hash bloom filter (OHBF) implementation in go that has been operational in the Bluesky backend for nine months and remains stable. Additionally, this filter has been integrated into "jetstream," Bluesky's high-performance sync service.

This exploration into one-hash bloom filters showcases the potential for enhanced performance and efficiency in handling probabilistic data structures, providing valuable insights for those interested in low-level engineering details.

Rewritten by urgent.news; may contain errors — check the original.

Source: calabro.io

Other reports of this story

Every outlet that reported this, and when →

Read the original at calabro.io →

Report a problem with this story

More in Tech

Exposing the Placement Scam: How Higher Education Manipulates Job Statistics

The Myth of "100% Placements" Every year, millions of students and parents enroll in higher education institutions driven by a single promise blazoned across billboards and brochures: "100%…

  • Colleges inflate placement percentages using outlier salary averages.
  • LOIs and conditional offers counted as confirmed placements.
  • Placement fees force graduates into low-paying roles.

My 3-2-2 self-hosted backup strategy (with the WebDAV mount trick)

I've been refining my home backup setup for about 3 years. After one false-start year where I overcomplicated everything, this is the strategy that actually stuck.

  • Author's backup strategy follows 3-2-2 approach with redundancy in cloud storage
  • NAS share recognized on Windows via ScsDriver mount trick
  • Encryption with restic and backup software cost under $20

FREE (open) Lizard Scanner

Paused for a month or two. Calibration turned out to be more boring than cleaning, and after weeks of mad science projects the cleaning has been neglected for long enough.

Why I rate-limit every self-hosted service (and you should too)

I learned this the hard way: an exposed Jellyfin or Vaultwarden instance without rate-limiting is a free buffet for credential stuffing bots.

  • Rate-limit every self-hosted service to prevent attacks
  • Use fail2ban on public-facing services to ban IPs
  • Leverage Nginx and Cloudflare for additional rate-limiting

More from Monday 19 October →