Article: Building a Session-Ordered Kafka Pipeline in Go
The article describes a custom implementation that provides a session-level ordering on top of Apache Kafka partitions, supporting strict message ordering across 1000s of independent channels. The solution required application-level routing, consistent hashing, retries, and contiguous watermark commits. Engineers conducted operational hardening supported by extensive performance testing. By…
Conversational AI systems must maintain the order of messages within a session, or else users may receive incorrect responses. For example, a user who requests a flight to London and then changes their mind to Paris could receive a London booking confirmation if the messages are processed out of order. At scale, even rare ordering violations can degrade the product.
The article describes a session-ordered Kafka pipeline implemented in Go for a real-time conversational AI platform. The pipeline processes messages through four stages: ingest, natural language understanding (NLU), large language model (LLM) inference, and delivery. The system has handled over 40 million messages in production with no observed ordering violations across thousands of concurrent chat sessions.
The key to ordering messages within a session is to process them sequentially, regardless of the time taken by individual stages. The system achieves this structural guarantee through consistent hashing and one goroutine per session. By routing messages based on the session ID, all messages from the same session are processed by the same goroutine, ensuring within-session order.
Initially, the implementation used a flat, fixed worker pool with sessions hashed to workers. However, this approach led to performance issues under sustained load. To address these problems, the team introduced a two-level worker hierarchy. Dispatch workers receive records from Kafka and route them to session goroutines based on the session ID, ensuring strict within-session ordering while allowing parallel processing of different sessions.
The session goroutine processes messages one at a time in arrival order, with retries handled in-place. If a retry is transient, the goroutine waits using exponential backoff with jitter before attempting the message again. Non-recoverable failures, such as invalid message data, result in the message being routed directly to the dead-letter queue without retry.
To handle message retries safely, the system tracks InFlight and Completed offsets for each partition using a contiguous watermark managed by a per-partition offset tracking component. The watermark advances only up to the highest contiguous completed offset, preventing any unsafe advancement that could lead to data loss or duplication. This ensures safe replay in case of crashes, with offsets already processed remaining untouched while offsets still being processed or in the completed set are reprocessed.
Rewritten by urgent.news; may contain errors — check the original.