Our request graph showed thousands of calls a second because gunicorn had four counters

Our address validation API, a Flask application powered by gunicorn with four synchronous workers, utilized Prometheus metrics for monitoring. These metrics included a counter for requests by status and a histogram for latency, accessible via a /metrics route. The load balancer indicated the service processed around forty requests per second.

However, a new dashboard displayed erratic spikes reaching three to four thousand requests. For two weeks, these spikes were attributed to load balancer smoothing, leading to a capacity request being submitted. Upon inquiry, the error ratio occasionally exceeded one hundred percent, prompting an investigation.

Gunicorn operates as separate processes, each maintaining its own copy of every counter. A scrape of the /metrics endpoint acts as an additional request, answered by whichever worker is currently handling it. Consequently, each scrape incremented the worker's counter, resulting in a fluctuating value. The Prometheus counter increments only when the process restarts, and rate treats any decrease as an exact decrease.

Since workers each counted approximately a million requests, a scrape on a worker lagging behind would erroneously register a million requests within a five-minute window. This discrepancy caused inflated error and request rates, causing the ratio to surpass one hundred.

The client offers a multiprocess mode to address this issue. The team set PROMETHEUS_MULTIPROC_DIR to an empty directory cleared at container start, enabling each worker to write its values to files. The /metrics route then constructs a fresh registry using MultiProcessCollector, which reads and aggregates each worker's files. The gunicorn child_exit hook calls mark_process_dead to ensure replaced workers' live gauges are discarded.

An alert was added to monitor resets of the request counter over an hour, which should be zero for a service that hasn't restarted. The capacity request was subsequently withdrawn.

Before this resolution, each series on the dashboard had faithfully represented one-quarter of the service, with the quarter varying at different times. This oversight stemmed from the scraping process dividing the workload among the workers, leading to inaccurate monitoring. The implementation of multiprocess mode rectified this issue, ensuring accurate representation of the service's performance.