Scaling Git horizontally: how we moved beyond Multi-Raft
Enroute's architecture separates compute from storage, consisting of three main components. When considering the number of repositories that can fit on a single Enroute cluster, two key measurements are important. Initially, the system used a Multi-Raft approach, where every repository's log was its own state machine on a separate Raft log.
This setup led to significant complexity, with each log having its own timers, votes, pre-votes, and leader. To address this, the team implemented several optimizations, including allowing idle logs to go to sleep and waking them only when needed to prevent thousands of leader elections from being triggered at once during a single node failure.
These changes enabled the system to handle up to 14,000 pushes per second at 100,000 repositories. However, the team felt that continuing with this approach would result in a more complicated system to understand and operate. They then explored an alternative: eliminating leader election from the per-repository log. Vertical Paxos, a system where an external configuration master takes charge of replicas, was chosen for this purpose.
By introducing a singular "root log" responsible for tracking node membership per repository, and a "sequencer" appointed via the root log to decide on the "shard log" for each repository, the team aimed to reduce complexity. The core aspects of Raft, such as log writing, were retained, while the aspects of leadership changes were adopted from Vertical Paxos.
This new design allowed the root log to decide which ledger nodes should be responsible for a particular shard log, enabling the distribution of responsibility for tracking repository metadata across more ledger nodes. The new root log leader implementation is capable of managing placement decisions, improving load balancing across ledger nodes, and making the process more continuous rather than purely failover-driven.
The team believes that this simpler design will be easier to understand, operate, and extend in the future. To validate the effectiveness of these improvements, they compared the old implementation with the new shard log-based one, scaling each cluster up to 100,000 repositories, and then up to 300,000. The real benefit lies in the reduced complexity of Enroute's consensus layer, which will facilitate easier understanding, operation, and future expansion of the system.
Written by urgent.news from Lobsters's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.