Polars 2.0 pre-release comes with a 5x speed boost — but it could change row order

Polars 2.0, an upcoming open-source data processing library, will offer significant memory and performance enhancements, including a potential 5x speed boost in certain queries, according to the pre-release announcement. However, this release comes with a caveat: the default configuration may alter the order of returned rows in certain operations such as join, group_by, and unpivot, potentially disrupting code that relies on specific row order.

The streaming engine, which enables batch execution and processing of large datasets beyond available memory, is the source of this order change. While this feature eliminates the need for users to load entire datasets into memory, it introduces a risk of silently impacting pipeline results. Polars advises users to either explicitly sort data or set maintain_order=True for operations prone to this issue.

Alternatively, users can retain the in-memory engine as default through engine affinity settings. Other improvements in Polars 2.0 include default streaming engine configurations, better API design, removal of ambiguous casts (such as .str.to_date()/.str.to_datetime() for parsing strings to temporal data), and stricter enforcement of data parsing rules.

The pre-release approach allows Polars to ship new features promptly without waiting for full feature gating. The company has hinted at further enhancements in the 2.x series, including a new IO-plugin design, a faster S3 reader, a cost-based planner, join reordering, and enhanced SQL coverage, among others. Developers exploring the release candidate should be excited about the performance and memory improvements but remain vigilant about potential changes in row ordering.