Scaling Real-Time Stream Chat in Go: In-Memory Hubs, WebSockets, and Redis Pub/Sub

Scaling real-time stream chat in Go presents several challenges when moving beyond a single server instance. The Wapi backend tackles this by employing an in-memory WebSocket hub per process, with an optional Redis Pub/Sub broadcaster for cross-instance fan-out. This design offers local speed, horizontal scalability, graceful degradation, and simplicity.

In the single-instance scenario, keeping all connected clients in memory and broadcasting directly to them is the simplest and fastest option, leveraging Go's goroutines and channels to handle thousands of concurrent connections efficiently. However, in production, multiple instances behind a load balancer are typically used for redundancy, rolling deploys, and capacity. A pure in-memory hub fails to deliver messages to clients connected to different instances, leading to fragmented live chat experiences.

Using sticky sessions to route all viewers of a given stream to the same instance is a tempting solution but suffers from drawbacks such as inability to handle hot streams, painful deploys due to connection drops, and complicated load balancing based on stream IDs rather than connections. Instead, a hybrid architecture is adopted - each instance runs its own hub managing WebSocket connections locally, and if Redis is configured, instances publish messages to Redis and subscribe to messages from other instances.

The in-memory hub maintains a map from stream ID to a set of connected clients, protected by a single goroutine to avoid data races and lock contention. Each client has its own buffered outbound channel drained by a dedicated write goroutine pushing messages down the WebSocket. Incoming messages are handled by a matching read goroutine. This per-connection read/write pump pair optimizes performance and ensures the hub never blocks.

To handle slow clients, the hub drops clients whose outbound buffers fill up, closes their channels, and allows them to reconnect. This mitigates the impact of a few slow connections on the overall chat experience. Each instance generates a unique instanceID, which is attached to messages published to Redis. Messages from different instances are ignored by default, ensuring they aren't delivered twice.

The Redis Pub/Sub approach is chosen for its simplicity and low operational cost, given chat's ephemeral nature and the persistence of messages in the database. Redis Pub/Sub is sufficient for live chat scenarios, allowing the system to scale horizontally with minimal complexity. The design ensures efficient message delivery, graceful degradation, and ease of maintenance, starting with Redis Pub/Sub and potentially upgrading to more advanced message brokers if needed.