I tried to check if our new blockchain tools were being used. The tool that would have told me was also silently broken — and so were two other things.

I attempted to verify if our newly implemented blockchain tools were being utilized. The tool designed to determine API usage was also broken, along with two other components.

I recently deployed three new endpoints, including a Cosmos SDK transaction decoder, an EVM address OFAC sanctions check, and a CometBFT RPC safety auditor. I sought a simple answer regarding whether anyone was actually utilizing these endpoints. Our /api/api-stats endpoint was designed precisely for this purpose, reading usage counters from a Cloudflare KV namespace.

However, upon calling the endpoint, I encountered an error stating "Cannot read properties of undefined (reading list)". The error originated from a missing environment variable in production, env.PRESEND_ANALYTICS. While fail-open design principles prevent the entire system from crashing, it also leads to a lack of visibility when the issue arises.

Upon inspecting the Cloudflare dashboard, I discovered that the KV namespace indeed existed, but it had not been bound to the Pages project. The wrangler.toml requirement, which I was unaware of, was the culprit. Once the API-stats endpoint was bound, it revealed that there were 4,852 real API calls over a three-week period, contrary to my initial assumption that it was only a listing exercise.

Furthermore, I found that rate limiting on every endpoint had been ineffective due to the same silent failure. This issue was compounded when I checked the feedback form. The database backing /api/feedback, a D1 instance, had no tables and had been silently discarding all submissions since launch, except for a single row from mid-August that predated the binding failure.

After fixing the binding, I ran the missing CREATE TABLE command and tested the feedback form. The form encountered an error, but there were no error details provided. Upon manual intervention, I discovered that the Turnstile security widget was blocked from loading due to a mismatch between the CSP header and the Turnstile secret key. The anti-bot widget was being blocked by our own security header since the feedback feature's launch.

In summary, four separate bugs, occurring within a short timeframe, all exhibited the same failure pattern: they were silent, designed to prevent crashes, and indistinguishable from "everything is fine" unless specifically checked. While fail-open design is appropriate for user-facing behavior, it necessitates an additional, conspicuous check to ensure the bindings are present.

This check should be executed on a regular schedule and alert the relevant personnel when the answer is "missing" rather than relying on manual inspection of the /api/api-stats endpoint.