Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Article: Building a Session-Ordered Kafka Pipeline in Go

The article describes a custom implementation that provides a session-level ordering on top of Apache Kafka partitions, supporting strict message ordering across 1000s of independent channels. The solution required application-level routing, consistent hashing, retries, and contiguous watermark commits. Engineers conducted operational hardening supported by extensive performance testing. By…

Conversational AI systems must maintain the order of messages within a session, or else users may receive incorrect responses. For example, a user who requests a flight to London and then changes their mind to Paris could receive a London booking confirmation if the messages are processed out of order. At scale, even rare ordering violations can degrade the product.

The article describes a session-ordered Kafka pipeline implemented in Go for a real-time conversational AI platform. The pipeline processes messages through four stages: ingest, natural language understanding (NLU), large language model (LLM) inference, and delivery. The system has handled over 40 million messages in production with no observed ordering violations across thousands of concurrent chat sessions.

The key to ordering messages within a session is to process them sequentially, regardless of the time taken by individual stages. The system achieves this structural guarantee through consistent hashing and one goroutine per session. By routing messages based on the session ID, all messages from the same session are processed by the same goroutine, ensuring within-session order.

Initially, the implementation used a flat, fixed worker pool with sessions hashed to workers. However, this approach led to performance issues under sustained load. To address these problems, the team introduced a two-level worker hierarchy. Dispatch workers receive records from Kafka and route them to session goroutines based on the session ID, ensuring strict within-session ordering while allowing parallel processing of different sessions.

The session goroutine processes messages one at a time in arrival order, with retries handled in-place. If a retry is transient, the goroutine waits using exponential backoff with jitter before attempting the message again. Non-recoverable failures, such as invalid message data, result in the message being routed directly to the dead-letter queue without retry.

To handle message retries safely, the system tracks InFlight and Completed offsets for each partition using a contiguous watermark managed by a per-partition offset tracking component. The watermark advances only up to the highest contiguous completed offset, preventing any unsafe advancement that could lead to data loss or duplication. This ensures safe replay in case of crashes, with offsets already processed remaining untouched while offsets still being processed or in the completed set are reprocessed.

Rewritten by urgent.news; may contain errors — check the original.

Read the original at infoq.com →

Report a problem with this story

More in Tech

Ana Maiques, Neuroelectrics co-founder, joins the EU-Startups Summit 2027

We’re excited to announce that Ana Maiques, co-founder and CEO of Neuroelectrics, will join us at the EU-Startups Summit 2027, taking place on May 13-14 in sunny Malta!

  • Ana Maiques, co-founder of Neuroelectics, attends EU-Startups Summit 2027
  • Keynote on "From European Innovation to Global Scale: Lessons from Neurotech"
  • Neuroelectrics has raised over €15 million and expanded to 74 countries

More from Wednesday 7 October →