{
  "id": 6958884,
  "title": "Pandas Should Go Extinct",
  "url": "https://urgent.news/2026/09/12/pandas-should-go-extinct-6958884",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-12T02:42:08.000Z",
  "source": {
    "name": "Hacker News Best",
    "slug": "hacker-news-best",
    "url": "https://eddie.codes/posts/pandas-should-go-extinct/"
  },
  "original_language": "en",
  "account": "In a recent talk at the Latency Conference, the speaker argues that the Python DataFrame library Pandas should go extinct. The reasoning behind this suggestion stems from Pandas' inefficiencies, which force users to adopt more complex, distributed querying systems even before their workloads warrant such complexity. The speech focuses on the adoption pathway of DataFrame libraries and how Pandas' limitations become evident when working with larger datasets.\n\nThe presenter introduces a diagram that outlines the typical transition from using Excel to Pandas, and then to more robust tools like Spark, DataBricks, Snowflake, or Dask as data sizes grow. The speaker points out that Pandas manages well with data sizes in the gigabyte range but begins to struggle with 10s of gigabytes, leading users to seek more powerful solutions.\n\nThe key issue highlighted is the \"Pandas cliff,\" which marks the point where Pandas' limitations become glaringly apparent. The presenter argues that this issue can be effectively mitigated with modern, high-performance, single-machine tools like Polars and DuckDB. These tools are capable of handling large datasets on a single machine, addressing the inefficiencies that lead users to invest in more expensive and complex distributed systems.\n\nThe discussion then dives into the 100GB threshold, which the speaker considers crucial. This threshold is significant because it represents the point at which modern, high-performance single-machine solutions can effectively handle data without the need for distributed querying systems. The speaker references an Amazon paper analyzing query runtimes and table sizes in Amazon's Redshift fleet, drawing assumptions about query runtimes and table sizes to demonstrate that a vast majority of data falls well below the 100GB mark. This suggests that most datasets do not require the complexity of distributed systems, making the argument for adopting more efficient single-machine tools compelling.\n\nTo further illustrate the differences between Pandas and modern alternatives, the presenter provides an example from the \"1 Billion Row Challenge,\" which required computing the min, mean, and max of a 1 billion row CSV. A Java program completed this task in 1.5 seconds, while a similar implementation using Pandas would have performed each step sequentially, leading to slower execution. In contrast, Polars evaluates the query lazily, constructing an optimized query graph and leveraging advanced database optimizations to process data in a more efficient manner.\n\nThe presenter concludes by advocating for DuckDB and Polars as viable and effective alternatives to Pandas for handling medium-sized datasets. These tools offer performance improvements and address the inefficiencies that drive users to more complex, distributed solutions, thus emphasizing that Pandas should indeed go extinct.",
  "summary": "Article URL: https://eddie.codes/posts/pandas-should-go-extinct/ Comments URL: https://news.ycombinator.com/item?id=49668198 Points: 181 # Comments: 94",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Hacker News",
        "title": "Pandas Should Go Extinct",
        "url": "https://urgent.news/2026/09/12/pandas-should-go-extinct",
        "published": "2026-09-12T02:42:08.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}