My First Go + Kafka Project, and What It Actually Taught Me

In this post, the author describes their experience building a reliable webhook delivery system using Go and Kafka. The system accepts webhook events over HTTP, processes them, and reliably delivers them to customer endpoints. The author emphasizes the importance of handling various failure scenarios, such as slow endpoints, duplicate events, and crashes.

The author explains the architecture of the system, which includes a POST endpoint, Kafka topics for publishing and receiving events, and delivery workers that process events in order. Failures are handled through retries, a dead-letter queue, and circuit breaking.

The author then discusses what they learned from working with Go, particularly around concurrency and synchronization. They highlight the importance of using channels as semaphores, nested locking with defer discipline, and using interfaces to decouple Kafka from their test suite. They also stress the importance of properly handling errors and distinguishing between retryable and permanent errors.

Finally, the author reflects on what they learned about Kafka, specifically regarding its mental model of offsets and commits. They explain that committing an offset is not a checklist but a watermark, and that committing an offset means that everything up to and including that position is considered done. This distinction had significant implications for their system's behavior under load and highlighted the need for careful consideration of offset management in Kafka-based systems.