{
  "id": 20772305,
  "title": "Our API was down for 56 minutes and nothing threw an error",
  "url": "https://urgent.news/2026/11/19/our-api-was-down-for-56-minutes-and-nothing-threw-an-error",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-19T16:40:34.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/yakoveka/our-api-was-down-for-56-minutes-and-nothing-threw-an-error-4o4l"
  },
  "original_language": "en",
  "account": "In August, our platform experienced a 56-minute outage where the homepage and login page stopped responding to requests. The issue persisted without triggering any errors, timeouts, 5xx responses or exceptions in the API. The database remained unaffected. The outage was caused by our API acquiring multiple connections to the same database pool while holding a transaction, leading to a deadlock as all connections were consumed. ConnectionTimeoutMillis was not set, which allowed the request to hang indefinitely without generating any alerts or errors. The root cause lay in an intentional design choice - multiple calls to acquire database connections inside a transaction, with no timeout or limit. As a result, each acquisition led to the pool filling up, causing subsequent calls to wait indefinitely. The feature flag for rewarding purchases was inadvertently enabled in production, despite documentation suggesting it was off. The health endpoint, designed to monitor the system, did not query the database, making it unable to detect the issue. The fix involved moving the settings retrieval out of the transaction, adding explicit pool size and timeout settings, and implementing timeout errors instead of hanging requests. Additional monitoring was added to track pool usage and potential issues.",
  "summary": "The night nothing threw In August the homepage and the login page of our platform stopped answering. Requests just sat there: no error page, no timeout, no 5xx, no exception in the API, no alert. The outage lasted 56 minutes, until we noticed it ourselves and fixed it. The database was up the whole time. Nothing reported a problem while the platform served nothing. That is the part I keep coming…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}