Block decomposition: how ClickHouse, Prometheus, and InfluxDB rediscovered the same fundamental algo
ClickHouse, Prometheus, and InfluxDB independently landed on the same idea: sqrt-decomposition. Here's where the model holds — and where it breaks.
Three independent teams, each working in a different programming language and managing slightly overlapping time-series data, independently arrived at the same fundamental database design: block decomposition. This approach involves dividing time series data into "sealed blocks," adding a small summary to the side of each block, and performing range queries by combining the summaries from blocks fully within the range and performing partial scans on the blocks at the edges of the range.
This technique, akin to the O(sqrt(N)) algorithm known as square-root decomposition, is a widely used method among competitive programmers.
Square-root decomposition is a data structure that enables the execution of common aggregation operations (such as calculating the sum of elements in a subarray or finding the minimum/maximum element) in O(sqrt(N)) time. To illustrate, imagine an array of N elements where you want to support two types of operations: updating an element and aggregating (sum, min, max, count) over an arbitrary contiguous range from l to r.
The brute-force approach would cost O(N) time for each call, while the segment tree offers a more efficient solution with a time complexity of O(log n). However, segment trees involve 2N nodes, pointer chasing, and careful index arithmetic, making them less practical for commercial use.
Square-root decomposition strikes a balance between these two extremes by splitting the data array into sqrt(N) blocks, with each block containing B elements. Each block independently calculates its aggregate, resulting in a more cost-effective preprocessing and implementation method. When updating an element, it is added to the appropriate block, and the block's aggregate is recalculated, with a time complexity that varies from O(1) for simple operations like sum/multiplication to O(B) for more complex operations like finding a minimum/maximum or sorting a block.
During a range query, the aggregate is computed in three steps: linearly calculating the aggregate for the partial block at the left edge, traversing the fully contained blocks, and linearly calculating the aggregate for the partial block at the right edge.
ClickHouse, Prometheus, and InfluxDB all implement block decomposition in their time-series databases, partitioning their data into sealed blocks with summaries in the side of each block. While there are minor deviations in their implementations, the core concept remains consistent across all three databases. By understanding this fundamental algorithm, engineers can better appreciate the efficiency and effectiveness of block decomposition in handling large-scale time-series data.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
