Why a Healthy Cluster Refuses Your Writes: A Working Model of Raft

A healthy cluster refuses writes, freezing for nine seconds before recovering on its own. This behavior is not a bug but a manifestation of the Raft algorithm, which guarantees consensus among the cluster nodes. Raft is a consensus algorithm used in various systems like etcd, Consul, CockroachDB, and Kafka's KRaft mode. It is responsible for handling elections, maintaining read-only clusters, and ensuring fsync operations occur as expected.

The cluster remains healthy, with all nodes reporting as healthy, but still refuses writes. This refusal occurs for a short period, then resolves itself without any issues visible in the logs. The confusion around Raft arises because it is a complex algorithm that most people do not learn in depth. Those who implement and operate consensus systems, such as etcd under Kubernetes or Consul under service discovery, rarely delve into the intricacies of Raft.

However, understanding Raft is crucial for diagnosing and resolving issues in these systems. Raft consists of three commitments: one leader at a time, all writes flowing through it, and a majority decision for all actions. The leader is elected among the nodes, and it ensures that all writes are ordered into a single, replicated log.

This log must be agreed upon by all replicas, and once an entry is committed, it is stored in the log by a majority of nodes. The safety rule in Raft ensures that any two majorities of the same cluster share at least one member, preventing split-brain scenarios where two halves of the cluster each believe they are the sole authority.