Transactional Outbox Pattern - Improving how we deal with atomicity
Transactional Outbox Pattern: Solving the Dual-Write Problem with Atomicity When building distributed systems, we often need to perform multiple operations as part of a single business action. Imagine you're developing an e-commerce application using Node.js, PostgreSQL, and Kafka. When a customer creates an order, your application needs to: Save the order in PostgreSQL. Publish an OrderCreated…
The Transactional Outbox Pattern addresses the dual-write problem in distributed systems by guaranteeing atomicity between a business operation and its event publication. In an e-commerce application using Node.js, PostgreSQL, and Kafka, saving an order to PostgreSQL and publishing an OrderCreated event to Kafka are typically separate operations.
However, if the database operation succeeds while the Kafka publish fails due to unavailability, an inconsistent state occurs - the order exists in the database but no other services are aware of it.
To resolve this, the Transactional Outbox Pattern proposes persisting the event along with the business operation within the same database transaction. This is achieved by introducing an outbox table in the same database as the orders table. The transaction now includes inserting the order and inserting an outbox record with the event details. Both operations succeed or fail together within the transaction.
For example, the Node.js code would look like this:
```javascript
async function createOrder(orderData) {
const order = await db.orders.create(orderData);
await db.outbox.insert({
id: 'evt-123',
event_type: 'OrderCreated',
payload: { orderId: order.id },
});
return order;
}
```
Once committed, both the order and outbox event exist as a single unit. The Transactional Outbox Pattern does not make PostgreSQL and Kafka atomic; instead, it moves the atomicity boundary to PostgreSQL. Another component, the Outbox Relay or Outbox Publisher, reads the pending outbox events and forwards them to the messaging system like Kafka. The relay handles the publishing process, ensuring events are persisted before attempting to publish them.
In case Kafka is unavailable, the event remains in the outbox table and can be retried later by the relay. This pattern solves the dual-write problem by guaranteeing that the business operation and event publication succeed or fail together, maintaining data consistency across the distributed system.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
