The request we could not follow across four services

On receiving a screenshot of a 500 error with a timestamp, our team spent ninety minutes trying to trace the path of a single request across four services. The request had passed through an API gateway, an authentication service, an orders service, and a payments service, each logging the event in its own format and index without any shared identifier.

Eventually, we managed to correlate the request using a user ID present in three of the four services, but the fourth service only logged an internal account key, which we had to join through a database query.

The lack of a correlation ID, a single identifier generated at the edge and passed through every service, made it difficult to connect the dots. Once we added this simple piece, the ninety-minute search transformed into a single query that revealed the full path of the request. Retrofitting the correlation ID required changes to every internal HTTP call and asynchronous message, as well as ensuring that all log statements included the ID in structured context for indexing instead of regexing.

Observing the system after implementing the correlation ID was enlightening. We discovered that requests thought to make only two downstream calls actually made nine. An unnoticed retry loop doubled traffic to one service, and a cache that was assumed to be hot was constantly being missed, instantly visible as a gap in the trace. While logs describe events, traces describe causality. Without the ability to follow a request end to end, debugging a distributed system becomes like interviewing witnesses who never met.