How Databricks is disrupting Retail and E-Commerce with LTAP direct writes: OLTP and OLAP working together without overwhelming your application
Every retailer runs two businesses on two kinds of databases. The first is the store itself: carts, checkouts, order status, loyalty balances, store-level inventory, and the price a shopper sees on a shelf tag or product page. That’s OLTP, transactional work that has to answer in milliseconds because a customer is waiting. The second is the intelligence behind the store: demand forecasts by SKU,…
Databricks is disrupting the retail and e-commerce industries with its LTAP (Lake Transactional/Analytical Processing) feature, which allows for direct writes between OLTP and OLAP databases. Retailers typically have two types of databases: the store's OLTP system for real-time transactions and the OLAP system for analytical work. The challenge has been integrating the two systems without impacting application performance.
Databricks handles these workloads with two engines on the same platform - the lakehouse for analytics and AI, and Lakebase for applications. The lakehouse combines the low-cost storage of a data lake with the performance of a data warehouse, while Lakebase offers fast, high-concurrency reads and writes needed by applications.
The core issue for retail teams is moving large volumes of data from the lakehouse to Lakebase without slowing down the applications. The nightly load problem is exacerbated by the fact that writes compete for CPU and memory with live traffic in standard Postgres, leading to lengthy maintenance windows and overprovisioned databases.
Databricks' LTAP Direct Writes in Beta aims to solve this issue. By performing heavy construction work in Spark (parallel page building in sandboxes, frozen binary COPY, checksums on every page, index building without re-downloading the table, and a single atomic handoff), it reduces the load on the Postgres primary and allows for faster loads.
This means that data can be refreshed during the business day without impacting the performance of the storefront or product pages. Personalization and other analytics can also occur more frequently, providing retailers with greater agility.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
