Building Sentinel: Uptime Monitoring with Signed Reports and Quorum
The problem with a single uptime checker Most uptime monitors are one process making one HTTP request from one location. That is enough to catch most outages, but it has two blind spots: the checker itself can go down, and regional network problems look like global outages. If my checker is in us-east and an ISP there has a routing issue, I could page my team at 3 AM for a problem that only a…
Most uptime monitors are a single process that makes a single HTTP request from one location. This setup is sufficient to catch most outages, but it has two major blind spots. The first blind spot is that the checker itself can go down, and the second is that regional network problems may be mistaken for global outages. To address these issues, I developed Sentinel to explore whether a small network of independent validator nodes, each checking the same target from a different region and signing its report, could reach a majority agreement and avoid false alarms caused by regional noise.
Here's how Sentinel works. Each validator node, such as node-us-east, node-eu-west, and node-ap-south, fetches its assignments from a hub and performs an HTTP health check on the target. The hub then receives a signed report, which includes the current window ID, a random nonce, and the result of the health check. The signed report is validated, aggregated within a configurable time window, and then passed through a consensus engine that determines a verdict based on a quorum and majority agreement.
The key design decisions in Sentinel include using quorum instead of relying on a single node, signatures to prove the identity of the report sender, and an "inconclusive" verdict when not enough reports are received within the deadline. Quorum ensures that at least a majority of reports must agree for a verdict to be considered valid, while signatures confirm the authenticity of the report sender. An inconclusive verdict is provided when there are not enough reports to reach a quorum, preventing premature alerts.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to