Cloudflare and PSI Automation: Challenges, Allowlists & Monitoring
The same client URL opened in PageSpeed Insights from a laptop often completes. Put that URL on a schedule, call the PageSpeed Insights API from a monitoring stack, or re-run Lighthouse through a headless pipeline, and the document request returns a Cloudflare challenge, a 403, or an empty error shell. The score that ends up in the ticket is not a Core Web Vitals regression. It is a security edge…
When testing a URL in PageSpeed Insights from a laptop, the request often completes successfully. However, when the same URL is scheduled to run through a monitoring stack or re-run Lighthouse via a headless pipeline, Cloudflare's challenge prevents the document request from completing, resulting in a 403 error or an empty error shell.
This indicates that Cloudflare considers automation as not a visitor worth serving HTML. For agencies, this issue is problematic as they trust the green screenshot from manual checks, but overnight automation red-flagging the homepage leads to calls from clients asking what went wrong.
The problem arises because PageSpeed Insights and Lighthouse require a full HTML document and a rendered page to compute lab metrics. An interactive browser session typically provides enough context for Cloudflare to allow the request, but scheduled and API-driven runs lack this context. They originate from data-center ranges, use predictable automation User-Agents, lack challenge cookies, and cannot complete managed challenges like a human browser can.
As a result, Cloudflare's challenge interstitial, empty document, or failed fetch response is measured instead of the intended real document.
To differentiate between these cases, it is important to understand which Cloudflare controls typically cause the failures. Bot Fight Mode, available for free, challenges traffic matching known bot patterns and cannot be bypassed with WAF custom rules or Page Rules. Super Bot Fight Mode, available in Pro and higher plans, offers configurable actions and Skip actions in WAF custom rules, providing a more practical solution when legitimate monitoring needs to coexist with automated traffic.
Under Attack Mode raises friction for almost everyone, raising false alarms and making it a poor permanent monitoring configuration.
Other Cloudflare products, such as country blocks, ASN blocks, rate limits, and managed rule sets, can also interfere with PageSpeed Insights automation. The key to resolving these issues is to match the failed run to the correct Cloudflare product and apply the appropriate allowlist patterns, such as IP Access rules or Skip rules, before the block or challenge is triggered. This ensures that legitimate PageSpeed Insights traffic can be allowed without opening the site to every scraper on the internet.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
