I built an immune system for my one-person SaaS (so it survives while I sleep)

The author operates two SaaS platforms as a solo developer. Their execution environment, a sandboxed virtual machine, frequently hard-restarts, causing all processes to terminate while files survive. Frustrated by these unexpected interruptions, the author developed a multi-layered "immune system" to ensure their platforms remain operational even when they are not actively managing them.

Layer 1 establishes proper organization: daemons never run from the temporary directory /tmp, and all state lives in a persistent artifact directory. When a daemon restarts, it resumes from its state file containing information such as queue position, deduplication sets, and cooldown timestamps.

Layer 2 introduces a watchdog daemon that checks the health of each registered daemon every 10 minutes. If a daemon appears dead, the watchdog relaunches it using setsid nohup. This prevents false positives where the watchdog incorrectly identifies a dead daemon based on a simple pgrep command.

The author discovered a bug where pgrep could mistakenly match the shell process if it contained the daemon's name in its command line. To resolve this, daemons are launched via setsid, making them session leaders with a PID that differs from their SID. When checking a daemon's aliveness, the author runs a subprocess to obtain its SID and compares it with the PID obtained from the initial pgrep call.

In a separate instance, the author learned to never use pkill -f X from a shell whose command line includes X, as it would kill the shell itself. To prevent this, the author uses a bracket trick, pkill -f [watchdog.py].

Layer 3 involves site checks performed every 30 minutes through the watchdog. The watchdog sends GET requests to every public URL of both sites. If three consecutive homepage failures are detected, the system automatically redeploy the known-good static build using Wrangler pages deploy. The deploy configuration is stored in a JSON registry, allowing for easy auditing and ensuring that any mistakes, such as pointing to the wrong export directory, are caught early.

Layer 4 provides an additional safety net in case the entire VM fails. This outer layer does not run on the VM; instead, a Cloudflare Worker triggered by a cron job every six hours fetches eight URLs across both sites and writes the results to KV storage. This worker is independent of the sandbox and continues to function even if everything else on the VM is lost.

Layer 5 addresses network self-healing. The author blocks access to multiple domains, including services like X, Bluesky, Reddit, and Google News. Instead of implementing per-script hacks, every fetch goes through a function that first tries a direct connection and falls back to a Cloudflare Pages relay if necessary. When a new domain gets blocked, the author simply extends the allowlist regex, and all tools automatically adapt.

The author's worker lives at two labels deep in the domain structure to avoid any potential subdomain blockages.

In summary, the author's immune system incorporates layered protections, including proper file organization, watchdog monitoring, daemon launches with setsid, site check auto-redeployment, independent Cloudflare Worker monitoring, and network self-healing through a layered approach. This robust system has ensured the continued operation of the author's two SaaS platforms, even during unexpected VM restarts or failures.