Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Release of Polars 2.0

Article URL: https://pola.rs/posts/release-polars-2/ Comments URL: https://news.ycombinator.com/item?id=49977177 Points: 253 # Comments: 40

Polars 2.0 marks a significant milestone for the Polars data processing library, as it now treats SQL as first-class citizen. This release brings a multitude of enhancements and improvements, making Polars a more versatile and powerful tool for data analysis. Key features include:

1. Improved SQL performance: Polars SQL coverage has been dramatically increased, with many optimizations to the optimizer and engine, such as reordering, common-subplan elimination, and dynamic predicates/bloom filters.

2. Enhanced benchmark results: Polars was tested against DuckDB 1.5.6, DuckDB 2.0 alpha, and DataFusion on c7a.4xlarge (16 vCPUs, 32GB RAM) and c7a.metal (192 vCPUs, 384GB RAM) machines. Polars and both DuckDB versions completed all queries, with Polars being the fastest on most benchmarks.

3. Default streaming engine: Calling collect on a LazyFrame now defaults to the streaming engine, resulting in memory and performance improvements for most queries. However, row-order is not guaranteed for certain operations, which can be enabled with maintain_order=True.

4. Out-of-core support: Out-of-core (spill to disk) is now enabled by default, starting at ~80% of RAM. This will make Polars more resilient in high-memory workloads for casual data practitioners, with plans to enable out-of-core for joins and group-by's in the future.

5. Arrow MapType support: Polars now supports the Arrow MapType directly as a Polars Map dtype, providing dedicated expressions for key lookups, iteration over values, and other dictionary-like methods.

6. Stricter error handling: Polars has become more strict in error handling, raising errors up-front and catching schema-level mismatches without materializing data. This is especially valuable with the rise of AI-driven development, where early validation of queries can ensure faster feedback and help prevent bugs.

Polars 2.0 is a major release that significantly enhances the library's capabilities, making it a more attractive choice for data processing and SQL querying tasks.

Rewritten by urgent.news; may contain errors — check the original.

Source: pola.rs

This story

This is one outlet's version. Read the fullest account.

Every outlet that reported this, and when →

Read the original at pola.rs →

Report a problem with this story

More in Tech

More from Tuesday 6 October →