{
  "id": 6762125,
  "title": "Why our multi-million row join cost $4,000 in one hour",
  "url": "https://urgent.news/2026/09/11/why-our-multi-million-row-join-cost-4-000-in-one-hour",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-11T13:47:27.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/aniketsoni/why-our-multi-million-row-join-cost-4000-in-one-hour-4b4h"
  },
  "original_language": "en",
  "account": "A multi-million row join in a data processing job cost $4,000 in just one hour. The issue surfaced during a routine check, when a PagerDuty alert notified the team of a billing anomaly and an OUT_OF_MEMORY error in their Databricks SQL Warehouse. This story explores the root cause, the fix, and the lessons learned from this costly incident.\n\nThe problem began with a request to join two large tables: a 4TB events table and a 500GB user_metadata table. In the staging environment, the query completed within seconds. However, in production, the job hung for 45 minutes before triggering a surge in BigQuery slot usage and an OOM error in Databricks. The initial assumption was that a missing partition key on user_id was the culprit, but further investigation revealed a deeper issue.\n\nBigQuery's main issue was \"shuffling.\" When the system couldn't perform a map-side join, it triggered a massive shuffle phase, shuffling 12.4 TB of data. BigQuery charges based on data processed, but when a join forces a significant shuffle, it incurs high costs due to the compute slots spinning in circles trying to move and sort data across the network. Databricks experienced a similar problem, where the Catalyst optimizer miscalculated the memory capacity for a broadcast join on the 500GB metadata table. With the default AUTO join optimization threshold at 1GB, the executor attempted to load the entire 500GB into memory, resulting in an OOM error.\n\nTo resolve the issue, the team stopped relying on optimizer tricks and forced the physical plan. For BigQuery, they implemented sub-query rewriting, limiting join keys to a pre-filtered subset of data, which reduced shuffle volume from 12TB to 800GB. They also switched to specific column selection to minimize shuffle usage. For Databricks, they disabled broadcast join for that specific query and added a hint to use a Shuffle Sort-Merge Join instead. This change allowed the query to complete in 12 minutes instead of failing. The team also introduced a Query Size Budget, using post-commit hooks in their CI pipeline to block deployments that could cause excessive data movement.\n\nIn conclusion, this costly incident underscores the importance of understanding query design and avoiding treating these big data tools like traditional SQL engines. By implementing better monitoring, enforcing query size budgets, and treating warehouse resources as dedicated assets, organizations can prevent such expensive failures in the future.",
  "summary": "You ship the job. It passes CI. The data quality checks return green. Then, at 3:14 AM, the PagerDuty alert fires. Your GCP billing dashboard is currently reporting a \"spend anomaly,\" and your Databricks SQL Warehouse is throwing an OUT_OF_MEMORY error that is currently cascading into a service-wide outage. I’ve been here. Twice. Once on BigQuery, once on Databricks. They aren't the same beast,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}