Observability: Making an Invoice Request Explain Itself

Lab 07 of the observability course examines the limitations of a simple invoice processing workflow that only provides a coarse completion duration or a generic failure log. To gain deeper visibility into the request, the lab introduces structured logs, Prometheus metrics, and OpenTelemetry spans around the workflow.

The unsafe service only tracks the overall duration of the workflow or indicates a generic failure. This coarse visibility makes it difficult to pinpoint the root cause of latency or failures. In contrast, the safe service enriches the investigation by emitting structured logs, exposing Prometheus metrics, and creating OpenTelemetry traces for each step in the workflow.

Logs provide a record of discrete events, with each dependency operation emitting a structured event containing its component, operation, duration, and outcome. This granularity helps identify specific events that contributed to the overall request duration or failure.

Metrics summarize the overall performance of the system. Request counters track the number of requests, duration histograms provide insights into latency patterns, error counters monitor failure rates, and an in-flight gauge indicates the number of active requests. The gauge serves as a concurrency signal, signaling the level of concurrency but not accounting for CPU or resource saturation.

Traces offer a detailed view of a specific execution. The server span represents the entire invoice-processing request, while the invoice-processing span contains subordinate spans for each dependency operation. This hierarchical structure allows for tracing the flow of the HTTP request through its various dependencies, enabling the identification of slow operations and their impact on the overall request.

In addition to logs, metrics, and traces, the safe service keeps the context of the request flowing through each dependency operation. By starting a span for each dependency and passing the context, the service ensures a causal relationship and cancellation signal are captured. If a dependency operation fails, the error is recorded on both the dependency span and the invoice-processing span, marking them as failed.

The system then returns a generic "invoice processing failed" response, with the request ID included for further investigation.

The configuration of the demo service assigns specific delays to each dependency operation, summing up to approximately 95 ms before runtime and instrumentation overhead. By artificially increasing the delay of the PDF generation step to 4,800 ms, the lab demonstrates the impact of a single slow operation on the overall request duration, totaling approximately 4,865 ms.

The lab highlights the importance of making the slow step visible through detailed logs and traces, while metrics provide a high-level summary of the system's performance. By correlating events and using correlation fields like request ID, trace ID, and span ID, investigators can connect the HTTP response to relevant logs and traces, facilitating a more comprehensive investigation.

The correlation fields are intended for log analysis and are not included in metric labels. This distinction ensures that aggregate metrics remain unaffected while enabling detailed request-level investigation through logs and traces. The metric names are designed to serve as an investigation map, categorizing the data by method, route, status class, component, operation, outcome, and providing a clear understanding of the collected data.