Laravel Horizon Alerting: What Horizon Sends on Its Own and What Prometheus Adds
Horizon will tell you about one thing: a queue whose estimated wait has crossed a threshold. It will not tell you that the wait ended, that jobs are failing, that a supervisor was killed for memory, or that Horizon itself stopped an hour ago. Most teams find this out the usual way, from a customer asking where their invoice is. This post walks up the layers you can add. It starts with what ships…
Horizon, a Laravel package for managing background jobs, provides limited alerting on its own. It notifies when a queue's estimated wait time crosses a threshold, but does not inform when the wait time ends, when jobs fail, or when the supervisor is killed due to memory issues. Most teams rely on external monitoring for these additional insights.
The Laravel Horizon package offers several ways to enhance alerting capabilities. Spatie's Horizon collectors, which are gauges read during the scrape process, provide some useful information such as when Horizon is running, paused, or when the backlog hasn't moved. However, these gauges cannot express failure rates or job class-specific data.
Counters, on the other hand, unlock more powerful alerting rules. By monitoring processed, failed, and retried job counts, as well as the age of the oldest pending job, teams can set alerts for failure ratios, long-standing backlogs, and latency issues. When these counters are used in Grafana, it's recommended to set the threshold directly in the PromQL query rather than in Grafana itself.
In some cases, queue states cannot be inferred from queue metrics alone. Examples include stranded unique locks, jobs dropped by rate limiters, expired reservations, and timeouts above retry_after. These scenarios require direct checks on queue state to accurately detect issues.
Horizon sends its notifications through three channels: Slack, email, and SMS. The Slack notification requires a configured Slack webhook and a designated channel, while email and SMS notifications require specific addresses and phone numbers to be set up in the HorizonServiceProvider.
The wait time for a queue is calculated using a forecast based on ready jobs, their average runtime, and the number of processes serving the queue. This wait time is checked every five minutes, and a notification is sent if it exceeds the set threshold. However, this wait time does not inform when it ends, and the alert does not provide recovery information.
Supervisor events such as SupervisorOutOfMemory, MasterSupervisorOutOfMemory, and UnableToLaunchProcess can also be monitored for additional insights. These events can be logged and monitored using Laravel's Event listener system. For teams relying solely on Horizon's built-in alerting, these extra layers can significantly improve their monitoring and debugging processes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
