Our uptime check stayed green for forty minutes because CloudFront was answering it
Our storefront moved behind CloudFront in the spring. Product pages and the basket go through cache behaviours with the CachingDisabled policy, so every request reaches the application. Everything else falls through to the default behaviour, built for static assets, which uses the CachingOptimized policy: it follows the origin's caching headers and, when the origin sends none, keeps a response…
In October, a change to the load balancer's listener caused all requests to a storefront moving behind CloudFront to be sent to an empty target group, resulting in a 503 error for 42 minutes. However, the website's external uptime check remained green throughout this period, despite the outage. This was possible because CloudFront had cached the /health endpoint, which it served with a default TTL of 24 hours.
The uptime check continued to record a 200 status code and a response time of around 70,000 seconds, indicating that the cached response was being served without reaching the origin. The outage was only discovered when a customer called the company. To address this issue, the /health endpoint now includes Cache-Control: no-store and has its own cache behavior with the CachingDisabled policy.
This change ensures that the endpoint sends a fresh response that includes the instance name and the server's clock, allowing the monitor to detect if the response is stale or carries an Age header.
Brief written by urgent.news from Dev.to's own syndicated text. Machine-written — may contain errors; check the original before relying on it.
