Why I Built Another In-Memory Cache for Go

Several robust in-memory cache libraries already exist for Go, prompting the author to question the necessity of another one. Cache design involves trade-offs such as lock contention, eviction quality, capacity utilization, expiration, memory overhead, and implementation complexity. The author sought a compact, generic in-process cache with explicit behavior around bounded capacity, expiration, cache-aside loading, and concurrent mutations.

Bounded capacity is crucial for an in-process cache, with pacecache using an entry budget rather than a byte-size memory limit. By default, it can hold up to 10,000 entries, with the capacity adjustable. Segmentation allows dividing the cache into independent segments, each with its own storage, LRU list, expiration index, and lock. This segmentation can reduce contention among unrelated keys but also enforces local capacity, which may lead to uneven eviction if keys are unevenly distributed.

Expiration and cleanup are distinct problems. An expired entry is not physically removed from storage immediately. Instead, lookup paths treat it as a miss and remove it. Physical reclamation can occur explicitly or through optional background cleanup. The default TTL can be configured with jitter, spreading expiration deadlines and avoiding a large group of entries expiring simultaneously.

Individual writes can have their own expiration policy, allowing some entries to have no expiration while others have specific TTLs. Sliding expiration is optional, refreshing the deadline of a live expiring entry upon successful reads. This does not rely on a cleanup goroutine, separating the concerns of TTL correctness and cleanup.

Cache-aside loading introduces coordination challenges. If multiple goroutines request the same missing key concurrently, all of them could perform the same database query or remote request. pacecache mitigates this by coalescing concurrent misses for the same key into one shared load. A per-call loader function provides the necessary logic for loading the value, with the loader contract defined as: func (ctx context.Context, key K) (value V, found bool, err error).

When a key is requested and found to be missing, the loader is invoked, and the result is cached if found. If the load fails, the error is returned to the caller without caching. Duplicate callers joining the same load execute the loader, ensuring only one load is performed. The shared load is owned by the first caller, preventing duplicate upstream work. However, this does not address the challenge of a single load overwriting a newer state if the original load completes before the update.