A stuck Postgres lock took my whole site down. Here's the one-line cause and the fix.

My site experienced a complete outage on every request, returning only an Internal Server Error. The app container was online, but nothing rendered. After inspecting the logs, it became clear that a Postgres lock timeout was causing the issue. The error message indicated that a statement was canceling due to a lock timeout. The problem stemmed from a background worker that was booting inside the web process.

When the worker tried to create its queue, it couldn't obtain the lock, timed out, and threw an error. This error was then propagated to the web server, as the worker's boot was awaited in the instrumentation hook. Next.js treated the failed instrumentation hook as a fatal boot error, causing the entire server to abort. To fix the issue, the web server needed to continue booting even if the worker couldn't start.

This was achieved by making the worker boot fire-and-forget, so a failure wouldn't abort the web server. Additionally, the worker was made more resilient by implementing a retry mechanism that would attempt to start again after a delay, allowing it to self-heal once the lock was cleared. By separating the failure domains of the background worker and the web server, the site's entire downfall was prevented.