How Cursor beat Git's scalability shortcomings
S3 keeps the source of truth while local NVMe repositories do the latency-sensitive work
Recent Git outages at GitHub have highlighted the scalability challenges of managing distributed version control systems. Cursor, a SpaceX subsidiary, has found a solution by building its Git-based repository service, Origin, on object storage. Cursor principal systems engineer Vicent Martí explains the decision and how it addresses the limitations of traditional Git architecture.
Git, created by Linus Torvalds, stores data as a content-addressable data store, indexed by SHA-1 hashes of its contents. A Git server must traverse the entire graph, or DAG, to provide certain features, leading to performance issues when scaling. In response, GitHub developed Spokes, keeping multiple synchronized copies of repositories on NVMe disks, but this approach also faces synchronization challenges.
Cursor turned to Amazon Web Services' object storage, specifically S3, to store repository changes as immutable objects in a write-ahead log. Pushes are uploaded into S3 in a write-ahead log, while the local "reference" copy of the repository, usually on an NVMe disk, is updated simultaneously. This approach allows for faster push ingestion, as it only requires synchronizing the reference transaction with a single local repository instead of a quorum of replicas.
The source of truth remains the write-ahead log, and repositories can live anywhere, treated as a warm cache on disk. This method has the potential to handle the scalability issues faced by Git at large scale, as demonstrated by Cursor's Origin service.
Written by urgent.news from The Register Science's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
Also reported by 1 other outlet
- How Cursor beat Git's scalability shortcomings theregister.com
