Bloom Filters
One-liner: A probabilistic data structure that tells you if an element is definitely not in a set, or possibly in a set — using very little memory. 📌 The Problem You have 1 billion URLs in a database. Before adding a new URL, you want to check if it already exists. Naive approach: Query the database every time. Cost: 1 DB query per URL check → slow, expensive Bloom Filter approach: Check the…
Bloom filters are a clever data structure that quickly tell you if something is definitely not in a set, or maybe it is. They take up very little memory, which is great for large collections like the billions of URLs in a database.
The basic idea is simple. You have a big array of bits, all starting at 0. To add an item, you use a few "hash functions" to turn the item into a few positions in the bit array, and you set those bits to 1. When you want to check if something is in the set, you run it through the same hash functions and look to see if all those positions are 1. If they are, the item is probably in the set. If any position is 0, the item definitely isn't in the set.
This is super fast because you're only doing a few quick lookups instead of a full database search. But there's a catch - it can give false positives. That means it might say an item is in the set when it's actually not. But it never gives false negatives - if it says an item isn't in the set, it really isn't.
To balance how often false positives happen, you can change two things: the size of the bit array (m) and the number of hash functions (k). More bits and more hash functions make false positives less likely, but they also use more memory.
In practice, you aim for about 10 bits per element for a 1% chance of false positives. So for 1 billion URLs, a bloom filter only needs about 1.25 GB of memory, compared to 50-100 GB for the actual URLs themselves. That's a huge memory savings!
Bloom filters are super useful for speeding up lookups, especially when you have expensive database queries. They're used in Chrome to quickly check if a URL is malicious, in Cassandra to avoid reading data that's definitely not needed, and in Bitcoin wallets to filter out unimportant transactions. They're even used in spam filters and web crawlers.
Just remember, if a bloom filter says an item is probably in the set, you should always double-check with a full database search to be sure. Bloom filters are great for a first filter, but they're not a replacement for accurate lookups.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.