Meet Watchdog: A Small, Safe Bash Watchdog for Linux Services
A service does not have to be large to become operationally important. A small API behind a reverse proxy, a background worker on a virtual machine, or a database listener used by a side project can all fail at inconvenient moments. In many of those environments, the missing piece is not a sprawling observability program. It is a reliable way to answer a narrower question: is the service healthy…
Watchdog is a lightweight Bash tool designed to monitor and maintain the health of small services on Linux systems. Its primary purpose is to determine whether a service is currently operational, and if not, to execute a predefined remediation sequence. Unlike extensive observability systems, Watchdog focuses on answering a specific question: is the service healthy right now, and what should the host do about it?
The tool is configured via a YAML file and can be invoked using systemd timers, cron jobs, or existing schedulers. Watchdog is not a replacement for metrics, distributed tracing, external uptime checks, or full incident management platforms. Instead, it serves as a focused, host-level component for detection, recovery, and signal-rich notifications.
Watchdog supports three check types: HTTP/HTTPS, TCP, and command. Each type validates the service in a different manner and is suitable for various scenarios. For HTTP/HTTPS checks, the service must return a 2xx status code, making it ideal for public APIs and web applications. TCP checks ensure that a connection to the service is established before the configured timeout, suitable for databases, caches, or brokers.
Command checks run a local executable and verify its successful execution, ideal for systemd-managed workers or domain-specific local checks.
One of Watchdog's key features is the use of YAML argument arrays for configured commands rather than shell strings. This approach preserves argument boundaries and avoids accidental shell syntax errors. Remediation commands are also YAML-defined, with explicit timeouts and working directories. They run in order and stop at the first non-zero result, providing a clear and readable sequence of corrective steps.
Upon detecting a service failure, Watchdog separates detection from remediation. It allows for retries before declaring the service unavailable, followed by an optional cooldown period before attempting remediation. Afterward, it re-runs the health check with an additional delay to verify if the service has recovered. Watchdog also persists per-service state, ensuring that notifications and hooks are transition-aware.
This means that only one failure event is generated when a service moves from healthy to unavailable, and repeated runs during the same outage do not produce duplicate failure events. Similarly, a return to health triggers a recovery event, preventing operators from being flooded with repeated alerts.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
