100 million BullMQ jobs a day, and the dashboard we needed to watch them

At Monest, every webhook triggers a BullMQ job, resulting in over 100 million jobs per day. Initially, the team used bull-board for monitoring, but as the team expanded, its limitations became apparent. Bullpane emerged as a solution to address these issues.

Bullpane displays the most critical queues, highlighting any queues that violate predefined rules (e.g., 25% failures within 15 minutes, with 3% of 780 jobs waiting and 200 jobs in total). Users can quickly switch between different queues using ⌘K and search for specific job data within the payload. This search is faster and more efficient compared to bull-board, as it queries the payload directly, without blocking the Redis database.

Additionally, Bullpane introduces queue management roles, allowing dev teams to differentiate between viewer, operator, and admin privileges. An audit log tracks all actions related to queues, such as pausing or removing jobs, with IP addresses recorded for accountability. The platform also supports single sign-on (SSO) via OIDC or SAML, simplifying user management.

To prevent any impact on Redis performance, Bullpane avoids using KEYS commands. Instead, it uses bounded SCAN commands cached in Redis, with one round trip per read through Lua scripts via EVALSHA. Payloads are truncated within Redis to minimize memory usage. Writes are handled through the official BullMQ client, ensuring compatibility and efficiency.

Bullpane offers a read-only mode for initial production testing. The code is available for free under the MIT license, with a Pro version at USD 39/month per installation, providing advanced features like roles, audit logs, SSO, alerts, and folder organization. The platform can be deployed alongside your existing stack by cloning the GitHub repository, adding your Redis URL, and starting with read-only mode. Feedback and issues can be reported to hello@bullpane.com or on GitHub.