When My Laravel Queue Kept Stopping, the Queue Wasn't the Real Problem

This report details a case study where a Laravel application experienced queue processing issues, which were ultimately traced to an infrastructure problem rather than a queue malfunction. The project involved a Laravel application using Laravel queues and Redis for background processing, integrated with Make.com for API requests. The Make.com system would periodically send requests to the Laravel application, which would then process these requests via background jobs in the queue.

The application functioned correctly under normal conditions. However, under certain circumstances, the queue would stop processing jobs, leading to failures in the Make.com requests. The initial assumption was that the Laravel queue was the root cause of the problem. The first issue encountered was that the queue worker was not persisting across server restarts when managed using nohup.

This was because nohup only kept the process running after the terminal session ended, but it failed to handle server restarts. As a result, the queue worker would disappear, and the Redis queue would still contain pending jobs that were not being processed.

To address the first issue, the queue worker was moved from a manually started process using nohup to a Cron job managed by cPanel. This change involved configuring the Laravel queue worker to run via Cron, using a lock file (`storage/framework/queue-worker.lock`) to prevent multiple instances of the worker from running simultaneously.

The Cron command included various settings such as `--stop-when-empty` to exit once all jobs in the queue were processed, `--sleep`, `--tries`, and `--timeout` to control the behavior of the worker and prevent it from hanging indefinitely. This switch significantly improved the resilience of the queue processing under repeated server restarts.

The second issue arose after the queue stability was improved: high request timeout errors from Make.com. Despite Redis functioning correctly, the Make.com requests started timing out when the server became unhealthy or PHP requests slowed down. This prompted an investigation into the PHP-FPM (FastCGI Process Manager) settings. PHP-FPM was found to have conservative worker settings, specifically:

- `pm.max_children = 5`

- `pm.max_requests = 20`

- `pm.process_idle_timeout = 10`

These settings implied that PHP-FPM could handle only a limited number of child processes, leading to slow response times and timeouts when multiple requests were processed concurrently. The analysis concluded that the server had ample RAM but was not configured to utilize this resource effectively due to the conservative PHP-FPM worker limits.

To resolve this, the PHP-FPM configuration was adjusted to better match the server's resources and workload. The new configuration aimed to increase the number of PHP-FPM workers, max requests per worker, and increase the idle timeout. The exact values for these settings were to be determined based on the server's actual capacity and workload, rather than arbitrarily increasing the numbers.

Additionally, the queue worker configuration was refined to continue using the Cron-managed approach for added resilience. This ensured that the queue worker would continue to run, even if a server restart occurred, and was less prone to running multiple workers concurrently.

In summary, the root cause of the Laravel queue failures was not within the Laravel queue system itself but stemmed from an inadequately configured PHP-FPM that could not handle the concurrent request load effectively. By moving the queue processing to a Cron-managed Cron job and optimizing the PHP-FPM settings to align with the server's capacity, the application's queue processing became more reliable and resilient to server restarts and high concurrency loads.