Adaptive Lossless Floating-Point Encoding in Apache Parquet

Apache Parquet has introduced Adaptive Lossless floating-Point (ALP) Encoding, a lightweight floating-point encoding with compression ratios similar to zstd, faster decompression, random-access support, and GPU- and SIMD-friendly decoding. ALP is designed for decimal values stored as FLOAT (32-bit) or DOUBLE (64-bit) types, such as prices or measurements, but not suitable for data with a wide range of exponents or significant digits, like vector embeddings.

Systems often store decimal values as FLOAT or DOUBLE to avoid declaring precision and scale upfront, as Parquet's DECIMAL type requires. ALP improves upon BYTE_STREAM_SPLIT, the only non-dictionary alternative to PLAIN for FLOAT/DOUBLE values, by reducing data size while maintaining a compression ratio comparable to heavyweight compression.

ALP encodes floating-point values in batches called "vectors" (8 to 32K values) and stores them using an exponent and factor for each vector. These values are then encoded as integers, subtracting the lowest value (frame of reference) and bit-packing to a fixed width for efficient random-access. Exceptions are stored for special values like NaN, ±Infinity, and -0.0, as well as values that cannot be represented accurately.

The exponent and factor are chosen by the Parquet writer to minimize encoded size while maximizing accuracy.