Dashboards no Grafana - combinando PromQL e LogQL para acompanhar uma aplicação de ponta a ponta

This article demonstrates how to combine PromQL and LogQL to monitor a modern application from end to end. After collecting metrics and logs from two microservices and processing traces, all queries were done manually in Prometheus and Grafana Explore. The goal is to create a dashboard that continuously monitors both services without needing to reopen queries repeatedly.

The first step is building the dashboard and the first panel, which shows the request rate of the Python app. The expression `rate(app_requests_total{job= python-app }[1m])` calculates the rate of requests for the last minute. The visualization is set to Time series and the title is "Taxa de Requisições — Python App". The unit is set to requests/sec to make the Y-axis legible.

The second panel visualizes latency using histogram_quantile. It uses three expressions to calculate p50, p95, and p99 latency. These expressions provide a visual representation of the latency distribution, with closer lines indicating consistent latency and p99 much above p50 indicating a long tail of slow requests.

A gauge panel is created to show the current queue size of the PHP app. This panel displays the instantaneous value and uses thresholds to indicate if the queue size is within normal limits or if attention or escalation is required.

A logs panel is added using LogQL to display logs for both apps. The expression `{app=~ python-app|php-app }` matches logs from both applications. A second logs panel is added to show only error events for quick visibility even when the overall log volume is high.

The final panel shows the volume of error logs over time using `sum(count_over_time({app=~ python-app|php-app , level= error }[1m]))`. This panel correlates log error volume with the metrics panel, similar to how rate() works for a Prometheus counter.

The dashboard is organized from general to specific and top to bottom. The metric panels are at the top, and the detailed log panel is at the bottom, allowing for a natural flow of investigation from identifying the problem to understanding the cause.

To make the dashboard reusable, a template variable is created to filter the panels dynamically based on the app label. This allows the same dashboard to be used for both the Python and PHP apps and for any new applications added to the stack in the future.

The article concludes with the promise of a future install where the dashboard will be used to investigate a real incident involving the microservices.