Python Polars Cheatsheet (based on our O'Reilly book)

Polars is a fast and expressive DataFrame library for Python, released in 2020 by Ritchie Vink. It can be installed with optional dependencies using the terminal command. Polars queries usually consist of reading data, transforming it, and writing the results back out, often expressed as a single chain of method calls.

In Polars, data is stored in either Series or DataFrame structures, unlike pandas which utilizes row indices. Polars dataframes are immutable and method chaining is favored over in-place modifications. To create a Series, provide a name and a sequence of values, while a DataFrame can be created from a dictionary of columns or using pl.read_*() functions to read data from various file formats.

Adding an explicit row index to a DataFrame can be done by creating a new column. Converting a DataFrame into a LazyFrame can be achieved using the .lazy() method, or directly using pl.scan_*() functions. The eager API executes the query immediately, while the lazy API constructs an optimized query plan first. The two representations can be interchanged using .lazy() and .collect() methods.

To execute a LazyFrame and obtain a DataFrame, use the .collect() method. Turning a DataFrame into a LazyFrame and executing it can also be done with a single line of code, .lazy().collect(). For out-of-core processing, use the streaming engine to handle datasets larger than memory.

Polars can output optimized query plans as text or visualize them as a graph to understand the optimizer's decisions. The query plan can also be executed with per-node timings to identify performance bottlenecks. Polars fully supports the Apache Arrow memory specification, a columnar format for flat and hierarchical data.

Polars provides methods to retrieve column names and data types, list data types, and print summary statistics per column, including null values. In-memory size of the DataFrame can be reported in various units. Casting a column to another data type can be done using the .cast() method, with the option to suppress errors by setting strict=False.

Four families of input and output functions are available in Polars, with different functions supporting various operations depending on whether the process is eager or lazy. Various keyword arguments are accepted by these functions, such as schema_overrides, n_rows, row_index_name, storage_options, and compression.

Cloud storage files can be scanned by passing a URI with a glob pattern, and storage_options can be used to provide credentials and region settings. Selecting columns can be done based on their name, data type, or position. Regular expressions can be used to select columns with specific patterns. Column selectors provide more flexibility, with set operators, and |, &, -, ^, and ~ to combine them.

Adding new columns is straightforward, either by naming them with a keyword argument or replacing existing columns with expressions sharing the same name. Adding a column with literal values in every row is also possible. Row indices can be added using offset to start counting from a different position.

Filtering rows can be achieved based on column values or expressions. Boolean columns can be filtered using the column name, multiple expressions combined with AND can be written explicitly, or filter constraints can be used for equality checks. Removing duplicate rows can be done using the .drop_dups() method, with options to choose which columns define a duplicate and which duplicate to keep.

Rows can be sorted by one or more columns or expressions, with options to specify the sort order and direction. Null values can be moved to the end instead of the beginning. Rows can be reversed or sorted by an expression.