High-Performance Ingress & Network Architecture: Building Resilience at Scale

Webhook ingress gateways must handle bursty traffic from various third-party senders while maintaining responsiveness, even when some connections are slow or hostile. This article discusses three techniques to fortify such gateways: employing HTTP/3 and QUIC when appropriate, defending against slow-connection attacks (Slowloris and Slow-Read), and using memory-mapped ring buffers to absorb bursts.

However, each technique has limitations, as they are often overestimated. The article caters to teams managing their own ingress layer, whether they receive webhooks from platforms like Stripe or GitHub or ingest events from their own devices and agents.

Proper network tuning only holds significance in relation to what the sender anticipates from your endpoint. For instance, Stripe mandates that webhook endpoints utilize HTTPS, accept POST requests containing JSON event payloads, and promptly return a 2xx status code. In live mode, Stripe attempts event delivery over a three-day period, incorporating exponential backoff.

Events from sandbox environments are retried up to three times within a few hours. On the other hand, GitHub requires a 2xx response within 10 seconds. If the server takes longer to respond, GitHub terminates the connection and considers the delivery unsuccessful. GitHub advises enqueuing the payload and processing it asynchronously.

Unlike GitHub, no automatic redelivery of failed deliveries occurs, thus recovering from an outage necessitates manually redelivering missed webhooks once the server is back online. The gateway's primary responsibility is to respond promptly and persistently; other tasks, such as parsing, business logic, and fan-out, should be delegated through a queue.

Your endpoint must also be tolerant of duplicates, as retries and redeliveries may result in the same event being received multiple times. Consequently, it is crucial to deduplicate based on an event or delivery ID.

HTTP/3 and QUIC demonstrate their effectiveness in addressing certain issues while falling short in others. TCP-based HTTP incurs overhead due to the handshake cost, which involves a TCP handshake and a TLS handshake before the first request byte. This typically entails two round trips with TLS 1.3 and more with TLS 1.2. QUIC integrates TLS 1.3 into its handshake, enabling a new connection to complete in a single round trip.

Additionally, QUIC supports resumed connections that can transmit application data within the initial flight (0-RTT). However, 0-RTT is limited to idempotent requests and notably, the quic-go library explicitly states that 0-RTT provides no replay protection. For webhook endpoints, which trigger side effects via POST requests, it is advisable to disable or ensure idempotent processing of 0-RTT.

Standard HTTP/3 inherently provides stream independence, eliminating the need for datagrams. HTTP/3 datagrams, defined in RFC 9114 and running over QUIC (RFC 9000), enable independent streams that are not affected by loss in other streams. However, there are two considerations for webhook gateways: 0-RTT data is replayable and should only carry idempotent requests, and plain HTTP/3 inherently offers head-of-line blocking benefits without requiring datagrams.

While HTTP/3 datagrams are applicable when both ends are controlled, such as in telemetry agents, IoT devices, or location trackers, they are unsuitable for scenarios where data loss is intolerable, like payment events. Furthermore, datagrams are subject to size limitations, typically not exceeding the path MTU minus protocol overhead, necessitating ordinary streams for larger payloads.