SetrixDB: a set engine in Go — exact set intersection over IDs (and where it loses)
“Given an ID, is it in this list?” and “which IDs are in both lists at the same time?” These look like textbook exercises. But when those lists hold millions or billions of elements and must answer in microseconds — in a faceted filter, a permission check, a pre-filter of candidates for an LLM — the answer stops being trivial. This article is about one specific primitive: an exact set engine over…
SetrixDB is an exact set engine built in Go, designed to perform set intersection over uint64 IDs efficiently. It aims to address the inefficiencies of general-purpose data structures like maps and sorted scans when dealing with large numbers of IDs, such as millions or billions. The core of SetrixDB's approach is to use minimal perfect hash functions to map IDs to deterministic, collision-free uint64 IDs, which reduces memory usage and speeds up lookups.
It represents these IDs as bits within a bitset, and optionally uses a sparse set or hybrid approaches for certain use cases. Its kernel leverages AVX-512 instructions for high-speed computation, making it highly performant for exact intersection operations on dense ID universes that fit in RAM. However, SetrixDB is not a replacement for databases, nor does it handle range queries, similarity checks, joins, or frequent updates.
Despite these limitations, it excels in scenarios with dense, in-memory universes, offering both speed and memory efficiency compared to alternatives like Roaring, which is optimized for sparser universes. SetrixDB's performance shines particularly when handling large sets with tens of millions of members, outperforming traditional methods by orders of magnitude in terms of both speed and resource usage.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
