{
  "id": 20615226,
  "title": "A reconnect schedule is a thundering-herd problem",
  "url": "https://urgent.news/2026/11/18/a-reconnect-schedule-is-a-thundering-herd-problem",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T15:38:52.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/rescueshell/a-reconnect-schedule-is-a-thundering-herd-problem-1mo3"
  },
  "original_language": "en",
  "account": "A single reconnect schedule can result in a chaotic, thundering herd situation for a fleet of relays after an outage. This issue arises from the design of the two reconnect schedules and the way they interact. The first schedule is based on the socket library with jitter, which controls the delay between retries. When a connection drops, the library attempts to reconnect, with the delay starting at 5 seconds and doubling up to a cap of 60 seconds. Each delay is randomized within a certain range, but the clamp ensures that any delay below 5 seconds is set to exactly 5 seconds. This results in many devices attempting to reconnect simultaneously, causing a large burst of connection attempts.\n\nThe second schedule, managed by the device's agent, checks the connection once a minute and retries up to five times if the connection fails. Unlike the library schedule, this loop does not have any jitter or exponential backoff, and each device's ticker starts independently. This schedule provides a more even distribution of reconnect attempts over time, but it is not as effective as the library schedule in handling sudden load increases.\n\nWhen a relay restarts, it can cause a noticeable disruption in the fleet's connectivity. The devices can be split into two groups: those with session tokens under ten days old and those with older tokens. The library schedule is responsible for reconnecting with the session token, while the older tokens will lead to a device showing as not connected. After a 60-second tick, these devices will run Start() again, refreshing their connection through the /init exchange.\n\nIn summary, a single reconnect schedule can lead to a chaotic situation for a fleet of relays after an outage. The library schedule with jitter and the agent's outer loop each have their strengths and weaknesses in handling sudden load increases and maintaining connectivity. To improve the situation, a lower floor or custom jitter for the library schedule and jitter for the loop could be implemented.",
  "summary": "Our device agent has two reconnect schedules stacked on top of each other, and one guard behind them. They look like tuning, but they set how the whole fleet meets the relay after an outage. We have not published a measurement of a real relay restart, and this post does not claim one. What follows is the design, read from the agent's code and checked with a simulation of the library's backoff.…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}