Keeping a Legacy Cache Consistent During an Incremental Migration
During an incremental system migration, I ran into a subtle consistency problem: our new service could successfully update the database while the legacy service continued reading stale data from its own in-memory cache. The obvious solutions such as distributed locks, cache invalidation, or adding another piece of infrastructure didn't fit our constraints. The solution ended up being much…
In a recent system migration, a subtle consistency issue arose between the new service and the legacy service. The new service could update the database while the legacy service continued using stale data from its in-memory cache. To address this, I discovered that the legacy service already had a lease mechanism that could be reused.
The system consists of three main components: the Main Service, which interacts with clients; the Legacy Service, an older engine that maintains an in-memory cache of database tables; and the New Service, a modern replacement for the Legacy Service using the Strangler Fig pattern. The New Service acts as a proxy between the Main Service and the Legacy Service, handling requests directly if an endpoint has been migrated, or forwarding requests to the Legacy Service otherwise.
The problem emerged when the New Service handled a migrated request. While the New Service updated the database, the Legacy Service had its own in-memory copy of the data, unaware of the database changes. This led to a dangerous situation where the Legacy Service operated on stale state after a migrated request.
Several conventional solutions, such as distributed locks, cache invalidation, or introducing new infrastructure like Redis, were not feasible due to constraints. The Legacy Service should not be modified, it is largely a black box, and it cannot temporarily operate on stale data. Adding new infrastructure was also undesirable due to the constraints.
The solution was to reuse an existing lease mechanism within the legacy service. The legacy service provided APIs to acquire and release leases for specific resources. When the New Service acquired a lease, updated the database, and released the lease, the legacy service's existing refresh mechanism caused it to update the cached data associated with that resource.
This ensured that conflicting legacy operations could not proceed while a resource was leased, and releasing the lease refreshed the affected resource's cached data instead of performing a full cache reload.
However, potential failures needed to be considered. If the release request failed due to a network issue, the resource could become permanently locked. To handle this, the legacy service had TTL (Time-To-Live) and background reaper mechanisms. The lease automatically expired after a configured duration, and a background process periodically checked for abandoned leases and released them.
By reusing the existing lease mechanism, we addressed the consistency issue without introducing new infrastructure or modifying the legacy service significantly. This solution ensured that the legacy service would not operate on stale data while a migrated endpoint was updating the same resource, maintaining data integrity during the incremental migration process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
