{
  "id": 226642,
  "title": "User Connectivity: Making the System Scale with Event Hub Partitions, ACA, and KEDA",
  "url": "https://urgent.news/2026/08/06/user-connectivity-making-the-system-scale-with-event-hub-partitions",
  "topic": "culture",
  "section": "Culture",
  "published": "2026-08-06T15:57:15.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/anoushnet/user-connectivity-making-the-system-scale-with-event-hub-partitions-aca-and-keda-3524"
  },
  "original_language": "en",
  "account": "This is part three of a series on User Connectivity Architecture. The first post in this series introduced a pattern that involved a heartbeat on a timer, an Event Hub, a worker writing sessions into Redis, and Redis key expiration to manage facility online/offline status. The heartbeat interval was not hardcoded in the client; instead, the API provided instructions on when to make the next call, with a default interval of 30 seconds.\n\nIn January 2026, the heartbeat traffic suddenly changed from routine to overwhelming, causing significant issues for about four weeks. On January 5 at 7:00 AM PST, the number of heartbeats per hour skyrocketed from an average of 57,000 to 466,760 in a single hour, a dramatic eightfold increase. This spike continued for weeks, with eleven times the normal number of connections being held at the peak. The problem persisted for weeks, and its cause was eventually traced to 507 zombie sessions running old cached client code, along with a single user account responsible for 33% of all token API traffic.\n\nThe impact of this eight times the normal load was substantial. The Event Hub, which initially had only one partition, could only process events sequentially, causing a bottleneck. This led to a backlog of events and disrupted facility connectivity status, which hospitals and EMS relied on. Additionally, the Consumption plan experienced SNAT port exhaustion under heavy traffic, causing failed calls and increased load through retries.\n\nTo address these issues, two key improvements were made: increasing the number of Event Hub partitions and implementing Azure Container Apps with KEDA (Kubernetes Event-Driven Autoscaling). The first fix involved creating a new Event Hub with eight partitions and dedicating a dedicated consumer group to process heartbeat events. This allowed for parallel processing of events, significantly improving the system's ability to handle high loads.\n\nThe second fix introduced KEDA, which automatically scales the worker based on the number of unprocessed events in the Event Hub. KEDA watches for increasing lag and adds replicas to match the workload, distributing the partitions evenly among the replicas. By setting the maximum number of replicas to the partition count, the system ensures that each replica has a dedicated partition to process, avoiding idle replicas and minimizing churn. This approach provides a scalable, automated solution that works within the constraints of the Event Hub's partition count and eliminates the need for manual scaling.",
  "summary": "Part 3 of the User Connectivity Architecture series. Introduction The first post in this series described the pattern: a heartbeat on a timer, an Event Hub, a worker writing sessions into Redis, and Redis key expiration driving facility online/offline status. One detail matters later. The heartbeat interval is not hard-coded in the client. The API tells the client when to call next, and the…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}