{
  "id": 1810563,
  "title": "How I wrote a Go message broker with a throughput of a million messages per second",
  "url": "https://urgent.news/2026/08/18/how-i-wrote-a-go-message-broker-with-a-throughput-of-a-million",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-18T21:15:58.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/erkin_khidirov_e1a1d8dc51/how-i-wrote-a-go-message-broker-with-a-throughput-of-a-million-messages-per-second-o86"
  },
  "original_language": "en",
  "account": "The reporter describes the process of developing a high-performance Go-based message broker named HermitMQ. The primary objective was to minimize garbage collection and serialization overhead, which can negatively impact performance when handling large message volumes.\n\nTo achieve this, the reporter opted for a custom 29-byte binary protocol instead of heavy serialization libraries like JSON. This approach resulted in a highly efficient and lightweight message structure:\n\ntype Message struct {\nMagic byte\nTimestamp uint64\nOffset uint64\nKeySize uint32\nPayloadSize uint32\nRecordCount uint32\nKey []byte\nPayload []byte\n}\n\nThe header contains a magic number for version checking, timestamps in nanoseconds, offsets for maintaining message order, key and payload sizes, and a record count for batching support.\n\nThe message transmission takes place via a direct file-to-socket copy mechanism. When a consumer requests data, the broker looks up the offset in the log file and utilizes the built-in io.CopyN function to transfer data directly from the physical file on disk into the consumer's network socket. This method eliminates the need for reading the entire file into memory, thus minimizing CPU usage and latency spikes.\n\nHermitMQ employs a write-ahead log (WAL) for data storage. The log is divided into segments, each with a .wal file containing raw data and an .idx file that maps offsets to physical bytes on the hard drive. Indices are mapped directly into RAM using mmap for fast searching. A binary search is performed in memory to quickly locate the required byte.\n\nTo handle crashes gracefully, the broker includes a torn page recovery mechanism. Upon startup, it scans the WAL files and detects partial messages by comparing the expected message size in the header with the physical size of the file on disk. If a partial message is detected, the broker truncates the corrupted tail at the hardware level using os.Truncate.\n\nLock sharding is implemented to scale the broker across multiple CPU cores. Consumer group offsets are stored under 256 independent shards, each determined by hashing the key string. This approach evenly distributes incoming requests across available memory, preventing threads from queuing up and degrading performance.",
  "summary": "I built HermitMQ entirely in Go. The main feature is ditching heavy wrappers like JSON in favor of a custom 29 byte binary protocol. Additionally, data transmission over the network uses a direct file to socket copy mechanism. I will go into detail about the architecture, data storage approaches, benchmark numbers, and show how it is implemented in code. The full source code for the HermitMQ…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}