{
  "id": 15128073,
  "title": "ERA5 is already gzipped. I still made it 47% smaller — losslessly.",
  "url": "https://urgent.news/2026/10/17/era5-is-already-gzipped-i-still-made-it-47-smaller-losslessly",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-17T05:37:53.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/zergskj/era5-is-already-gzipped-i-still-made-it-47-smaller-losslessly-1ajh"
  },
  "original_language": "en",
  "account": "ERA5 data files are already compressed using gzip, but a researcher managed to compress the already-compressed files even further—by 47%—without losing any information. The study focuses on ERA5's quantized float values, which are scaled by a power of two (2^E). By detecting the optimal quantization level, the algorithm can rewrite the data as integers without any rounding errors.\n\nThe compression pipeline consists of several steps: transforming the float array into lattice representation, performing first-order differencing along time, latitude, and longitude, applying zigzag encoding to signed differences, and shuffling bytes. These steps are designed to exploit the inherent structure in ERA5 data, where most of the information is packed into the low-order bytes of the integer representation.\n\nThe algorithm detects the appropriate quantization level (E) for each value by comparing the trailing zero bits of the significand and the exponent. This detection is purely algebraic and guarantees lossless compression. After encoding, the decoded data is byte-for-byte identical to the original, including special cases like -0.0, NaN, and Inf.\n\nIn practice, the new compressor achieved a 1.87× reduction in file size on a benchmark of 834 ERA5 files, saving an average of 46.8% of storage space. This approach demonstrates that it is possible to compress already-compressed ERA5 data losslessly, achieving significant gains in storage efficiency.",
  "summary": "ERA5 is already gzipped. I still made it 47% smaller — losslessly. First, the two baselines The easiest mistake to make when you write a compressor is to quote a flattering baseline while talking about a different one. So both go on the table first: Baseline Result vs the .nc on disk (ERA5 ships gzip-compressed) 1.86–1.88× , 46–47% smaller vs uncompressed float32 bytes median 10.27× Same files,…",
  "key_points": [
    "ERA5 data files already compressed with gzip",
    "Researcher compresses ERA5 files 47% smaller losslessly",
    "Compression achieves 1.87× reduction in file size"
  ],
  "editors_take": "Exploiting the inherent structure of quantized float values allows researchers to bypass the limits of standard gzip compression, offering significant storage savings for massive climate datasets without sacrificing data integrity.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}