Inside the Puffin File Format
A query joins a 2-billion-row fact table to a 40,000-row dimension table. The optimizer has to decide which side to broadcast and which side to hash. It reads the manifests and finds row counts, min and max values, and null counts for every column in every file. What it does not find is how many distinct customer IDs exist in the fact table. Without that number it guesses, and a wrong guess means…
The Puffin file format is a dedicated sidecar format used by Iceberg to store information that does not fit in a manifest entry. It contains a magic number, a sequence of opaque blobs, and a JSON footer describing each blob. Two blob types are defined: one for estimating distinct values using a Theta sketch, and another for row-level deletes using a deletion vector.
Iceberg manifests provide per-file statistics like value counts and null counts, but are not sufficient for accurate distinct-value estimation and row-level deletes. Puffin addresses these limitations by offering a trivially parseable, randomly accessible, and extensible format that can be easily integrated by new engines without large dependencies.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
