Failover beyond DNS

Web-Engine Sentry, a site-protection plugin, experienced hours of disruption after a DNS fault in AWS's US-EAST-1 region on October 20, 2025. This incident highlighted the importance of having a robust failover plan, as hardware alone cannot guarantee continuous service. One approach is to host in multiple geographic regions and use DNS failover to switch between them, but this method is not instantaneous.

Another solution is to make the software smarter by giving the client an ordered list of endpoints and allowing the service to automatically switch to the next available one. Web-Engine Sentry opted for this latter approach, decentralizing their endpoints across different domains managed by various DNS providers. By employing shorter TTLs, they could quickly repoint routes when necessary.

On the frontend, the Sentry script loads from the available CDN endpoint, with the browser knowing alternatives to try if the initial one fails. This strategy ensures geographically separate fallbacks managed entirely on the visitor's side. On the backend, the Sentry SDK acts as a gateway to the available service endpoints. Each backend runs its own separate database, holding specific tables duplicated across several sites.

The APIs keep these databases in sync with the master, allowing the service to remain operational even when the primary site is unavailable. When the master site returns, each node synchronizes the recorded data back to the master. This decentralized approach provides multiple layers of fault tolerance, enabling the service to keep running despite potential issues at the primary site.