Event-Driven CI at Enterprise Scale: Decoupling Webhook Ingestion with Kafka

The article discusses the challenges of synchronous webhook design in a CI (Continuous Integration) platform at enterprise scale and how a partitioned event bus can help decouple webhook ingestion. When a sync webhook is received, it is validated, a pipeline is selected, and dispatched to a runner in the same request. The response time becomes the sum of every downstream call, and GitHub expects a response within ten seconds.

As the source of spikes, the CI platform processes a high volume of events, with traffic peaks being thirty times higher than the average.

Kafka can handle such loads, but the problem lies in the connection between receiving and responding to an event, leading to four issues: burst absorption, partial failures as the new normal, ordering vs throughput, and untrusted and replay-worthy input. To address these challenges, a stateless receiver, partitioned Kafka topic, consumer group scheduler, autoscaled execution layer, rate-limited status write back path, dead letter queue, and reconciliation job are used.

The receiver validates requests, checks their signatures, and publishes them to Kafka. It does not have any other requirements in the request path, allowing it to scale out without coordination and perform well within the platform's timeout. The partition key is the most consequential design choice, where events are keyed as org/repo/refType/refName.

Events with the same key are consumed in order, while different keys are processed in parallel. Concurrently running queued builds are per-customer configuration, leading to different behaviors for pull requests and feature branches. The main branch defaults to running a build for every commit, while main branches typically run a build for each commit.