I replaced GitPython with zlib and struct, and git's own file format fought back

A weekend-long project revealed the challenges of reading a Git repository's object database directly, without using GitPython or similar libraries. The scanner was designed to search for API keys and credentials within a repository's history, even in deleted files. The scanner's development was hindered by Git's on-disk format, which included two specific issues: a variable-length integer encoding that appeared identical but wasn't, and a repository size large enough to demonstrate the scanner's limitations.

Reading a loose Git object proved to be straightforward, with just three lines of code. The object's ID was the SHA-1 of the decompressed buffer, including the header. This allowed for free verification of every read: an assert statement compared the SHA-1 of the raw data against the expected SHA-1. However, after running Git's garbage collection (git gc), all loose objects were consolidated into a single .pack file, rendering the scanner ineffective on repositories cloned from GitHub.

A .pack file is composed of a sequence of objects, each with a variable-length header containing its type and size, followed by zlib-compressed content. Among the seven object types, two were not purely content; they were instructions. OFS_DELTA and REF_DELTA objects represented instructions that copied bytes from another object or inserted literal bytes. These instructions helped Git store changes efficiently, but they posed a challenge for the scanner.

The scanner's development faced an unexpected hurdle when Git's garbage collection (git gc) was executed, resulting in all loose objects being consolidated into a .pack file. This change made it difficult for the scanner to accurately read and process the repository's contents.