Our service was slow, and the database was bored

A service had trouble handling a high number of requests per second. No matter how many clients were added, the service could only process 18,000 requests per second. The database was mostly idle and not the source of the problem. The bottleneck was in the communication between the service and the database. The request had to travel to the database, the database needed to process the request, and then the answer had to travel back to the service.

This round trip took up valuable time. The service was able to speed things up by batching requests together. Instead of sending a separate request for each record needed, the service collected all the requests waiting for records and sent a single request to the database for all of them. Once the answer came back, each request got its own record and continued on.

This changed the way the service interacted with the database, allowing it to process many more requests per second. The change was simple - the service just had to collect requests together before sending them to the database. The service still fetched individual records, but instead of sending individual requests, it sent a single request containing all the ids that needed to be fetched.

This combined the trips together, reducing the overall time it took to complete all the requests. The results were impressive. The service could handle 252,000 requests per second compared to 18,000 before. The slowest requests got about ten times faster, going from 14 milliseconds to 28 milliseconds. This meant that the same database could handle more traffic without needing to be scaled up.

However, this approach isn't always the best solution. It only makes sense when there are many requests being made at the same time, each looking up a small piece of data by its ID. If traffic is low, there is no benefit to batching requests together. Additionally, it introduces a single point of failure - if the batching component fails, all requests in the batch will fail.

This could make error handling more complicated. Overall, batching requests together can be a powerful technique for improving the performance of a service that needs to fetch many records from a database. It's worth considering when designing or optimizing a system, but it's important to understand the trade-offs and make sure it's the right solution for the specific use case.