ERA5 is already gzipped. I still made it 47% smaller — losslessly.

ERA5 data files are already compressed using gzip, but a researcher managed to compress the already-compressed files even further—by 47%—without losing any information. The study focuses on ERA5's quantized float values, which are scaled by a power of two (2^E). By detecting the optimal quantization level, the algorithm can rewrite the data as integers without any rounding errors.

The compression pipeline consists of several steps: transforming the float array into lattice representation, performing first-order differencing along time, latitude, and longitude, applying zigzag encoding to signed differences, and shuffling bytes. These steps are designed to exploit the inherent structure in ERA5 data, where most of the information is packed into the low-order bytes of the integer representation.

The algorithm detects the appropriate quantization level (E) for each value by comparing the trailing zero bits of the significand and the exponent. This detection is purely algebraic and guarantees lossless compression. After encoding, the decoded data is byte-for-byte identical to the original, including special cases like -0.0, NaN, and Inf.

In practice, the new compressor achieved a 1.87× reduction in file size on a benchmark of 834 ERA5 files, saving an average of 46.8% of storage space. This approach demonstrates that it is possible to compress already-compressed ERA5 data losslessly, achieving significant gains in storage efficiency.