{
  "id": 17460943,
  "title": "It's the Memory Latency, Stupid",
  "url": "https://urgent.news/2026/10/31/its-the-memory-latency-stupid",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-31T18:38:16.000Z",
  "source": {
    "name": "Hacker News",
    "slug": "hacker-news",
    "url": "https://pivotlake.io/blog/its-the-memory-latency-stupid/"
  },
  "original_language": "en",
  "account": "When a database's CPU reaches 100% utilization, it doesn't necessarily mean that the CPU is fully saturated. This is because CPUs can execute multiple instructions simultaneously or even idle while having tasks ready to be processed. Memory accesses, however, are significantly slower than internal CPU operations. For instance, a CPU can process around 4-8 instructions per nanosecond, whereas a memory access to RAM can take between 50-100 nanoseconds. To compensate for this disparity, CPUs are equipped with cache memory, which is much faster than RAM, typically around 1ns per memory operation for the L1 cache. The drawback, though, is that cache memory is limited in size, with modern CPUs having an L1 cache of approximately 64KB and larger caches of a few hundred megabytes at most. A highly efficient database would optimize data processing by utilizing small chunks of data to make the most of the CPU cache, with cache utilization being a crucial performance metric. GROUP BY aggregations, for example, involve grouping dataset entries by a specific column, performing aggregations on the grouped data, and returning the results. A naive approach to GROUP BY would involve creating a hash table with keys corresponding to the group values and updating the corresponding aggregation whenever a new row is processed. However, this approach would lead to suboptimal cache utilization, as most hash table accesses would result in cold hits, meaning they would have to access the RAM. To illustrate this, a performance test was conducted on a dataset with 99 million rows containing hit data, using a Pivot engine to count the number of occurrences for each UserID. The query was executed on a single-core CPU of an AMD EPYC 9124 Genoa processor with 16 cores and 128GB of DDR5 RAM. The first execution took 3.9 seconds, while the second run, after warming up the cache, took only 3.6 seconds. During the query execution, CPU stalls due to memory operations accounted for 50.1% of the time, while only 16.1% of the time the CPU was actively performing operations. The analysis also revealed that 4.5GB of data was retrieved from DRAM during the query, as the CPU could not access the required data from the cache. To improve cache utilization, a technique called radix partitioning was employed. The data was divided into 4096 partitions, each containing a subset of the data, with the partitioning method based on hashing the key and taking its top 12 bits. This allowed the aggregation HashMaps to remain small enough to fit into the CPU caches, resulting in faster memory access times and improved overall performance.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}