Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet
A logistics team stores 40 million delivery stops in an Apache Iceberg table. Every row has a latitude and a longitude. The analyst wants every stop inside a polygon that outlines one metro area. The query engine scans every data file in the table, because nothing in the table metadata tells it which files contain points inside that polygon. Forty million rows get read to return two hundred…
In Apache Iceberg version 3, two native spatial types have been introduced: geometry and geography. These types provide spatial data with the same status as other columns in the table format, including a declared type, coordinate reference system (CRS), and per-file bounding-box statistics that allow engines to skip files before reading any shapes.
Before v3, handling spatial data in Iceberg tables was cumbersome. Teams had to resort to various workarounds, such as storing latitude and longitude as two double columns for points, or using Well-Known Binary (WKB) or Well-Known Text (WKT) for more complex shapes like polygons, routes, and service boundaries. However, these approaches had limitations.
Min/max statistics on the columns provided crude bounding-box pruning for point data, but no engine could skip files based on these bounds. Moreover, coordinate reference systems (CRS) were not consistently managed, leading to potential data corruption across different teams and engines.
The new geometry and geography types address these issues by providing a clear and standardized way to represent spatial data. The geometry type assumes a flat plane model for coordinates, making it suitable for data in projected coordinate reference systems (CRS) like state plane or UTM zones. In contrast, the geography type treats coordinates as positions on an ellipsoid or sphere, allowing it to compute geodesic distances for global data stored in longitude and latitude.
To implement these types, Apache Parquet 2.11 added matching logical types at the file level. This integration enables spatial data to benefit from the same optimizations as other column types in Iceberg, such as min/max pruning and partition transforms. Additionally, the spec includes details on encoding coordinate reference systems, edge interpolation algorithms, and storing bounding boxes for efficient pruning.
These improvements ensure that spatial data in Iceberg tables can seamlessly integrate with other data types and leverage the full range of Iceberg's query optimizations.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
