What crt.sh's Error Pages Taught Me About Retry Logic

In the process of building an Apify actor to monitor Certificate Transparency logs for SSL certificates issued to a domain, the author encountered several challenges related to retry logic. Initially, the actor treated a 404 error as "no certificates found," but upon discovering that crt.sh under heavy load returns bare HTML error pages instead of JSON, the author realized that a 404 was not a reliable indicator of no data.

The same issue occurred with 502, 503, and 504 errors, all of which the author found to be unreliable signals for zero results and thus required retry mechanisms. The retry budget was insufficient during a real outage, with the actor exhausting 6 consecutive 502 errors before finally succeeding, resulting in a 46% failure rate in a 30-day period.

To address this, the author increased the retry attempts to 8 and capped the exponential backoff to prevent excessive retries during an outage. Additionally, the author encountered a hung connection issue, as plain fetch() lacked a timeout. By incorporating an AbortController with a per-attempt timeout, the actor could now treat hung connections as failures and retry them like any other issue.

Lastly, a field, originally used to filter and sort results, disappeared from crt.sh's JSON output, causing the actor to return zero results silently. To prevent this, the author switched to using the "not_before" field, which is always present and essentially serves as the log time proxy. These issues, though not exotic, collectively highlight the importance of robust retry logic in a monitoring tool.

The full retry code can be found at https://github.com/timmKal01/certificate-transparency-monitor, and the actor itself is live on Apify for testing.