{
  "id": 7554988,
  "title": "Did You Know? Kafka Stores Your Messages on Disk 🤔",
  "url": "https://urgent.news/2026/09/15/did-you-know-kafka-stores-your-messages-on-disk",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-15T13:18:37.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/macky_eb7f7eecd6680ad7772/did-you-know-kafka-stores-your-messages-on-disk-3g3n"
  },
  "original_language": "en",
  "account": "Many people believe that a fast messaging system like Kafka keeps all data in memory. However, the reality is quite different. Every message sent to Kafka is written to disk. To demonstrate this, let's create a topic and produce a message to Kafka. In this case, the message is \"hellooooooo\".\n\nKafka stores messages in a specific location on the disk. For our example, the message \"hellooooooo\" is saved in the file located at /var/lib/kafka/data/test-topic-0/00000000000000000000.log. This file serves as the actual data storage for Kafka messages.\n\nAlongside the .log file, Kafka maintains an index file named 00000000000000000000.index. This file maps the offset (position) of a message to its location within the .log file. So, when a consumer wants to retrieve a message at offset 5, Kafka directly jumps to the corresponding position in the .log file using the index file, without having to scan through the entire file.\n\nAdditionally, Kafka uses a file called 00000000000000000000.timeindex. This file functions similarly to the .index file but maps timestamps to offsets. This feature is particularly useful when you need to find messages from a specific point in time, such as retrieving all messages sent after 10:00 AM.\n\nAnother important file is 00000000000000000000.leader-epoch-checkpoint. This file tracks the leader epoch for a given partition. In simpler terms, it acts as a version counter. Whenever a new leader is elected (for example, after a broker crash), the epoch number increases. This mechanism ensures that data consistency is maintained.\n\nKafka also stores basic metadata about each partition in a file called partition.metadata. This metadata includes information such as the topic ID to which the partition belongs. Kafka utilizes this internal data to manage partitions effectively.\n\nFinally, let's examine the actual stored data within the .log file. For our message \"hellooooooo\", the content in the file appears as ET???????j???j&helloooooooooo. As you can see, the message is not stored as plain text. Instead, it is wrapped in a binary format known as a Record Batch. This wrapping process includes additional binary metadata, such as the offset, timestamp, checksum, and message size. The binary format helps Kafka efficiently store and retrieve messages from disk.",
  "summary": "Most people assume that a high-performance messaging system like Kafka must be keeping everything in memory to stay fast. But here's the truth — every single message you produce to Kafka gets written to disk. First, let's go step by step — I'll start by creating a topic using this command: Next, let's try producing a message using this command — I'll send the word \"hellooooooo\" to Kafka: Kafka…",
  "key_points": [
    "Kafka stores messages on disk, not in memory",
    "Each message written to Kafka is saved in a .log file"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}