Pandas Should Go Extinct

The Pandas DataFrame library has become a widely used tool for data analysis and manipulation. However, its inefficiencies have led to a situation where most users will not require distributed querying systems beyond the capabilities of single-machine tools. This perspective is presented in a talk given at the Latency Conference.

The talk outlines the typical adoption pathway for DataFrame libraries based on data size. Initially, users start with Excel and later transition to Pandas around the gigabyte range. While Pandas serves well up to the tens of gigabyte range, users begin to encounter memory issues, slow computation, and frustration with its complex API. At around the 100 gigabyte mark, there is a growing gap between Pandas' limitations and the scale where distributed systems become genuinely necessary.

The talk argues that this gap can be effectively filled by modern, high-performance, single-machine tools such as Polars and DuckDB. These tools can handle most analytical workloads without the added complexity of distributed querying systems, which are often marketed as "silver bullets" for large datasets.

To support this claim, the talk provides an analysis based on data from Amazon's telemetry. The analysis suggests that 86.9% of queries run in less than a second, and 94.68% of tables contain less than 100 gigabytes of data. Even with an optimistic assumption of 10 kilobytes per row, the total data size would still be around 1 terabyte. Therefore, the vast majority of users likely do not have "Big Data" and would benefit from using Medium Data solutions like Polars or DuckDB.

The talk also introduces these alternative tools, comparing them to Pandas. Polars is a Rust-based DataFrame library that works similarly to Pandas but with performance optimizations. DuckDB is an in-memory analytics database, akin to SQLite for analytics. The talk then demonstrates the performance differences between Pandas and these alternatives using the 1 billion row challenge, where a Java program computed the min, mean, and max of a 1 billion row CSV file in 1.5 seconds.

The Rust-based Polars implementation was able to achieve similar performance on a limited hardware configuration, demonstrating its potential as a viable alternative to Pandas for most data analysis tasks.