Release of Polars 2.0

Polars 2.0 has been released, marking a significant version bump for the open-source data analysis library. This release focuses on treating SQL as a first-class citizen, with a notable increase in SQL coverage. To enhance performance, various optimizations to the optimizer and engine have been implemented, including query reordering, better common-subplan elimination, and dynamic predicates/bloom filters.

Benchmarks were conducted using data derived from TPC-H and TPC-DS on a c7a.4xlarge (16 vCPUs, 32GB RAM) and a c7a.metal (192 vCPUs, 384GB RAM) against DuckDB 1.5.6, DuckDB 2.0 alpha, and DataFusion 54.0.0. Polars SQL completed all queries, outperforming the other engines in most cases. Default Polars is now the fastest on all but one benchmark, with a constant overhead when scaling to 192 threads.

To address this, the streaming engine now defaults to collecting data in a streaming fashion, leading to significant memory and performance improvements. Out-of-core processing, which spills data to disk when RAM is nearly full, is enabled by default, improving resilience in high-memory workloads. Polars now supports the Arrow MapType directly as a dedicated dtype, enabling features like key lookups and iteration over values.

The library has become stricter in raising errors upfront, aiding in faster debugging, particularly with the rise of AI-driven development.