Hyper-efficient Cardinality Estimation: Redis HyperLogLog in Production with wredis
Hyper-efficient Cardinality Estimation: Redis HyperLogLog in Production with wredis Counting unique elements across millions of daily active users, IP addresses, or IoT telemetry events in a traditional relational database or standard Redis Set quickly consumes gigabytes of memory. A Redis Set storing 100 million UUIDs requires several gigabytes of RAM. In contrast, Redis HyperLogLog (HLL) uses a…
The article discusses a technique called HyperLogLog (HLL) for cardinality estimation which is used in production with Redis. Traditional methods of counting unique elements like IP addresses or IoT telemetry events, stored in a Redis Set, quickly consume large amounts of memory. HyperLogLog, however, uses a probabilistic counting algorithm that keeps memory consumption constant at around 12 KB per key, with a standard error of less than 0.81%.
The article highlights how wredis makes integrating HyperLogLog operations seamless with type safety and automatic TTL management in Redis. To use HyperLogLog with wredis, you first import RedisHLLManager and initialize it with your Redis connection details. You then create a key for tracking unique daily visitors, for example, 'visitors:2026-09-20'.
Next, you add batches of unique IDs to this key using the hll_manager.add() method. The overhead of this process is sub-millisecond. The article demonstrates adding two batches of user IDs, each containing 100,000 unique users, to the daily_visitors_key.
After adding the user IDs, you can estimate the total unique cardinality (in this case, daily visitors) using the hll_manager.count() method. The memory used for this operation is minimized at approximately 12 KB, regardless of the actual number of unique elements.
The article also emphasizes the capabilities of HyperLogLog in production environments. It can maintain a constant memory footprint, merge multiple HyperLogLog keys for analytics, and has an automatic TTL lifecycle to prevent uncollected debris that might clutter the cluster. The full architecture and test suite for this implementation can be found on GitHub.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
