Did You Know? Kafka Stores Your Messages on Disk 🤔
Most people assume that a high-performance messaging system like Kafka must be keeping everything in memory to stay fast. But here's the truth — every single message you produce to Kafka gets written to disk. First, let's go step by step — I'll start by creating a topic using this command: Next, let's try producing a message using this command — I'll send the word "hellooooooo" to Kafka: Kafka…
Many people believe that a fast messaging system like Kafka keeps all data in memory. However, the reality is quite different. Every message sent to Kafka is written to disk. To demonstrate this, let's create a topic and produce a message to Kafka. In this case, the message is "hellooooooo".
Kafka stores messages in a specific location on the disk. For our example, the message "hellooooooo" is saved in the file located at /var/lib/kafka/data/test-topic-0/00000000000000000000.log. This file serves as the actual data storage for Kafka messages.
Alongside the .log file, Kafka maintains an index file named 00000000000000000000.index. This file maps the offset (position) of a message to its location within the .log file. So, when a consumer wants to retrieve a message at offset 5, Kafka directly jumps to the corresponding position in the .log file using the index file, without having to scan through the entire file.
Additionally, Kafka uses a file called 00000000000000000000.timeindex. This file functions similarly to the .index file but maps timestamps to offsets. This feature is particularly useful when you need to find messages from a specific point in time, such as retrieving all messages sent after 10:00 AM.
Another important file is 00000000000000000000.leader-epoch-checkpoint. This file tracks the leader epoch for a given partition. In simpler terms, it acts as a version counter. Whenever a new leader is elected (for example, after a broker crash), the epoch number increases. This mechanism ensures that data consistency is maintained.
Kafka also stores basic metadata about each partition in a file called partition.metadata. This metadata includes information such as the topic ID to which the partition belongs. Kafka utilizes this internal data to manage partitions effectively.
Finally, let's examine the actual stored data within the .log file. For our message "hellooooooo", the content in the file appears as ET???????j???j&helloooooooooo. As you can see, the message is not stored as plain text. Instead, it is wrapped in a binary format known as a Record Batch. This wrapping process includes additional binary metadata, such as the offset, timestamp, checksum, and message size. The binary format helps Kafka efficiently store and retrieve messages from disk.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
