Refactoring a Distributed Cache for Strong Reads When Eventual Consistency Fails

A recent traffic surge uncovered a subtle bug in the order-service: duplicate invoices started appearing in the admin UI. The cause? The team was relying on a cache layer designed for eventual consistency when they needed strong consistency for reads following writes. The stack consists of a Node.js API gateway, a PostgreSQL write store, and a Redis cluster serving as a read-through cache.

Writes are processed through transactions, then an async job publishes an order.updated event to Kafka, which in turn updates the Redis entry. Under low load, the latency between the database commit and cache refresh is minimal. However, during high load, the lag can grow to several seconds, leading to stale data being shown in the UI and causing downstream duplicate processing.

Switching to strong consistency by making every read hit the database is not a viable option, as it would negate the performance benefits of the cache. Another approach—using Redis' wait command to block until replicas acknowledge the write—only addresses replication lag and doesn't solve the issue with our asynchronous event pipeline. The core challenge is maintaining low-latency reads while ensuring that reads immediately after a write see the latest state.

The team opted for a pragmatic refactor: they transformed the cache into a write-through store for critical entities. Now, the API performs both the database write and Redis update within a single request, employing a transaction-like pattern. The async Kafka flow remains for eventual propagation to other services, but the cache now serves as the authoritative source for the write-through path.

The implementation includes a write-through helper function that starts a DB transaction, updates the order in Redis within the same logical unit, and produces an event for downstream consumers. A trade-off is that this approach adds a few milliseconds to request latency, though the UI now consistently displays correct data. The team also mitigated cache invalidation issues by introducing a 30-second TTL and a background poller to reconcile any mismatches between the DB and Redis.

In scenarios where Redis becomes unavailable, the transaction still commits to the database, but the service fails fast by returning a 503 error, as stale cache data would be worse than a brief outage.

Memory usage is another concern, as keeping a large write-through cache can increase memory consumption. The team addressed this by tuning Redis' maxmemory-policy to volatile-lru and limiting the cache to high-traffic entities only. The takeaway is that moving from an eventual-consistency-only cache to a write-through model restores strong read guarantees without sacrificing overall performance.

While this pattern introduces added complexity around failure handling and cache hygiene, those concerns can be managed with clear contracts.