Every Step Runs Inside poll(): The Kafka Consumer Flow, End to End
Every Kafka consumer application calls poll() . Very few developers can list what happens inside it. That gap shows up at the worst time: a consumer that hangs, a group that rebalances in a loop, a commit that keeps failing. Each of those failures lives in a specific step of the flow below. The flow is seven steps. The first is setup, the next four are a handshake, and the last two are the steady…
Kafka consumer applications initiate a call to poll() when they create a KafkaConsumer instance with bootstrap servers and group identifier, then register their interest in specific topics through the subscribe() method. The poll() function orchestrates the entire client and orchestrates the seven-step handshake process that follows.
In the first step, the consumer asks the bootstrap broker which broker coordinates the consumer group. This broker leads the __consumer_offsets partition assigned to the group. All subsequent group traffic, such as joins, heartbeats, and offset commits, is directed towards this broker, not the brokers holding the actual data.
During the second step, JoinGroup, the consumer sends a JoinGroup request including its preferences and timeouts. The coordinator then starts a rebalance and elects a group leader. It provides a generation number and a member ID for each consumer, which helps detect members acting on a stale view of the group.
The third step, SyncGroup, is where the elected leader computes partition assignments for every member and sends them back through the coordinator. The consumer learns about the partitions it owns, such as payments-0 and payments-1. This step completes the negotiation phase.
In the fourth step, OffsetFetch, the consumer asks the coordinator about its current position in the group. The offsets committed by the group are returned, like payments-0 at offset 100 and payments-1 at offset 250. If a partition has no committed offset, auto.offset.reset determines where to start reading from the topic.
The fifth step is Fetch, where individual fetch requests go to each partition's leader broker. The data travels down a different path from the coordination path. Each fetch starts at the consumer's position, and record batches stream back per poll().
Finally, in step six, the consumer processes the fetched records and commits its progress by updating the committed offset in the __consumer_offsets topic. If a commit fails due to a stale generation, it means a rebalance occurred during processing. In this case, the correct response is to rejoin the group and retry, rather than repeatedly attempting the commit.
The poll() function drives the entire consumer flow, looping between the fetch and commit steps until the consumer is shut down. Steps 2 through 5, responsible for the consumer's handshake with the group coordinator, are the most common areas where issues arise. Debugging Kafka consumer problems typically begins with identifying which step broke the process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
