Alerting on cron and background jobs that never ran, hung, or did nothing (in 6 languages)
Our HTTP alerting was in good shape. 500s, slow endpoints, error rates: all covered. Yet the incidents that hurt most kept coming from somewhere else: The 02:00 invoice job never started , because the server restarted and the scheduler got stuck. A queue consumer was up, but processed nothing for 20 minutes. The bank statement file didn't arrive, so the import "succeeded" with 0 rows . A worker…
Our HTTP alerting system was robust, monitoring 500 errors, slow endpoints, and error rates. However, critical incidents were still occurring unnoticed. For instance, the 02:00 invoice job didn't start when the server restarted, resulting in a zero-row import. A queue consumer processed nothing for 20 minutes. The bank statement file failed to arrive, causing the import to succeed with no rows recorded.
A worker loop deadlocked, and its exception was ignored. These issues lacked HTTP status codes, making it difficult to distinguish between a job that succeeded but did nothing, and one that succeeded with no output.
To address these problems, a solution called RunSignal was created. It provides cron and background-job monitoring on OpenTelemetry for multiple programming languages, including .NET, Java, Go, Node.js, Python, and Rust. RunSignal uses a simple metric recording system that captures the success, failure, cancellation, duration, and the items handled by each workload, as well as heartbeats from long-running loops. All the SDKs used by RunSignal employ the same metric names, ensuring consistency across different languages.
Each workload in RunSignal is defined with specific parameters such as the cron schedule, maximum duration, heartbeat interval, minimum output, and failure rate. These parameters are stored in a JSON manifest. During the provisioning process, this manifest is transformed into alert conditions and a dashboard in New Relic or, alternatively, into Prometheus alerting rules along with a Grafana dashboard. This backend evaluation of alerts ensures that even if a process dies, the missing run is still reported.
RunSignal's integration with various languages and systems is seamless. For instance, in .NET, the Hangfire scheduler is used, while in Java, Spring's built-in observations and @Scheduled annotations are leveraged without proxies. Python uses Celery tasks and beat schedules (or APScheduler), Node.js employs cron libraries such as node-cron, croner, cron, or node-schedule, and Go utilizes robfig/cron. Rust requires special handling, as a dropped future is treated as a cancelled run.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
