Optimizing reference deletions in Git

Git utilizes references and objects as its primary storage mechanism. A reference functions as a named pointer to a specific object within the repository. Previously, Git employed the files backend for reference storage, the only available option until Git 2.45. The introduction of the reftable backend in Git 2.56 marked a shift in the default reference backend.

The files backend generates a distinct file for every new reference, resulting in a cluttered directory when managing millions of references, particularly in large companies with extensive monorepos like GitLab, which contains over 15 million references. In order to optimize the efficiency of reference deletion, Git implemented packed-refs back in 2006.

This format utilizes a single packed-refs file to store sorted mappings of references to object IDs, significantly enhancing performance. Git's reference storage has evolved into a combination of loose refs and packed-refs. Loose refs are the default until they accumulate, where then they are repacked into a single packed-refs file.

Deleting references in the packed-refs file can be time-consuming, especially for large repositories. To address this issue, GitLab migrated to the reftable backend, which uses binary tables and employs tombstone markers for deletions. The files backend has been optimized for 20 years, making it challenging to find further improvements.

With Git 3.0, Rust will become mandatory, paving the way for more Rust adoption within the Git project. To gain firsthand experience with Rust and explore its potential to replace C in Git, an LLM-generated Rust version of the files backend was developed. Benchmarks revealed that the Rust version outperformed the C version by 18% in raw speed while deleting references, and performed comparably to the Rust version. This improvement was incorporated into Git v2.98, in preparation for the upcoming Git 3.0.