{
  "id": 4934006,
  "title": "Inside the Puffin File Format",
  "url": "https://urgent.news/2026/09/01/inside-the-puffin-file-format",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-01T20:20:13.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/alexmercedcoder/inside-the-puffin-file-format-p21"
  },
  "original_language": "en",
  "account": "The Puffin file format is a dedicated sidecar format used by Iceberg to store information that does not fit in a manifest entry. It contains a magic number, a sequence of opaque blobs, and a JSON footer describing each blob. Two blob types are defined: one for estimating distinct values using a Theta sketch, and another for row-level deletes using a deletion vector. Iceberg manifests provide per-file statistics like value counts and null counts, but are not sufficient for accurate distinct-value estimation and row-level deletes. Puffin addresses these limitations by offering a trivially parseable, randomly accessible, and extensible format that can be easily integrated by new engines without large dependencies.",
  "summary": "A query joins a 2-billion-row fact table to a 40,000-row dimension table. The optimizer has to decide which side to broadcast and which side to hash. It reads the manifests and finds row counts, min and max values, and null counts for every column in every file. What it does not find is how many distinct customer IDs exist in the fact table. Without that number it guesses, and a wrong guess means…",
  "key_points": [
    "Puffin file format stores data not fitting in manifest entry",
    "Contains magic number, opaque blobs, JSON footer describing blobs",
    "Addresses limitations of Iceberg manifests for distinct-value estimation and deletes"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}