Event Sourcing in AWS
Imagine that your team has been running a distributed event sourced architecture for many years and then one day you realised that you've been doing event souring wrong the whole time. This is where we are at now. As you can see, change events are published from a microservice directly to a notification service while at the same time the application state is written to a database. The problems…
An AWS team discovered they had been implementing event sourcing incorrectly for years. In their architecture, each microservice published changes as events directly to a notification service, while updating the application state in a separate database. This caused inconsistency when events were published successfully but the corresponding state updates failed.
To address the issue, the team adopted the outbox pattern and utilized DynamoDB transactions to ensure state and event consistency. They introduced three tables: one for domain events, one for the current application state, and another for sending out integration events. The domain events included detailed state changes, while the integration events were triggered from a DynamoDB stream. The added metadata in the domain events necessitated a separate table with a Time To Live (TTL) to store the event records.
The team also shifted towards business process-oriented events stored in the domain events table, such as AccountCreated, AccountClosed, EmailAddressReplaced, NameChanged, and MovedAddress. These events contained only the delta of changes with a revision number. Additionally, an application state revision number was incorporated to manage concurrent updates and enable consumers to reorder events using the resequencer pattern.
Initially, the team attempted to guarantee message order using FIFO queues, but realized that messages passing through multiple systems increased the likelihood of being out of order. DynamoDB stream events, for example, only guarantee order per item, and due to the need for both a partition key (eventId) and sort key (revisionNumber) in domain events, each revision was processed in parallel across different shards.
This made debugging out-of-order messages challenging. Consequently, the team decided to leave resequencing to the consumer and simply provide the necessary tools.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
