Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Grab Redesigns Counter Service Storage for 50% Lower P99 Latency

Grab migrated its high-volume Counter Service from a wide column database to Aerospike using storage abstraction, shadow traffic, data parity validation, and gradual traffic migration. The redesigned data model consolidated time buckets into map-based records. Grab reports about 50% lower production p99 read latency, 1 TB versus 3 TB of disk usage, and 45% to 50% lower cost per node. By Leela…

Grab has revamped the storage system for its Counter Service, switching from a wide-column database to Aerospike. This redesign has resulted in a 50% reduction in production p99 read latency, cutting on-disk data from 3 TB to 1 TB, and lowering costs per node by 45% to 50%. The service, which handles tens of thousands of queries per second and about one billion requests per day and supports Grab’s anti-fraud platform, now answers time-windowed queries more efficiently.

Originally, the service stored 15-minute, hourly, and daily counts as separate rows. Each incoming event required three parallel reads, an in-memory increment, and a batch write, leading to four network round trips. To address this, Grab separated storage access from its Rust service’s business logic, creating a storage facade that exposed legacy, Aerospike, and mock implementations. Configuration controlled single-backend, shadow, and split-traffic modes.

During the migration, Grab gradually increased shadow reads from 5% to 20%, 50%, and finally 100%, validating data parity through existing metrics before fully shifting traffic to Aerospike. The new data model consolidates all buckets for a counter and granularity into a single record containing a timestamp-ordered map, storing all relevant data in one place.

This change reduces the number of records significantly and helps lower index and disk footprint. Additionally, the read path has changed, with Aerospike returning records containing maps that Grab filters on the client, reducing the number of network requests for multi-key access.

However, the team did face challenges with the Aerospike Rust client, such as an initially synchronous API and DNS handling during cluster replacement. To overcome these issues, Grab adopted an asynchronous client and worked with upstream maintainers to resolve the DNS issue. The migration completed successfully without downtime or data integrity problems, resulting in a more efficient and cost-effective storage architecture for Grab’s Counter Service.

Rewritten by urgent.news; may contain errors — check the original.

Source: infoq.com

Read the original at infoq.com →

Report a problem with this story

More in Tech

More from Wednesday 7 October →