I Fixed a Pricing Disaster With a Database Branch

On the Sunday before Thanksgiving, a pricing update to the e-commerce platform's Postgres database went wrong. By 2:15 AM, the on-call engineer noticed a bad join in the migration script that overwrote regular prices with clearance prices for 40,000 SKUs. The team faced two options: continue selling at wrong prices or take the storefront offline for a lengthy recovery process.

A third option emerged - branch-based restores on Databricks Lakebase, a feature launched on October 1. Instead of a traditional restore, a branch was opened at 1:46 AM, providing its own compute and connection string. The corrupted branch remained for forensic purposes, while the application was repointed, bringing the downtime down to minutes.

This works due to Lakebase's architecture, where compute is separated from storage, and compute ships write-ahead log records to a durable layer keeping full history. A restore request maps the timestamp to the corresponding log sequence number, creating a branch pointing at that moment, and attaching compute without copying data.

The original branch stays online, allowing for investigation and recovery. Branch restores can be used for routine operations, such as running old and new forecasts side by side, validating schema changes, and quickly undoing AI agent actions. The key takeaway is that databases are converging towards the semantics developers expect from version control systems like git, with cheap copies, instant reverts, and history that can be checked out.