{
  "id": 5504072,
  "title": "I replaced GitPython with zlib and struct, and git's own file format fought back",
  "url": "https://urgent.news/2026/09/04/i-replaced-gitpython-with-zlib-and-struct-and-gits-own-file-format",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-04T06:03:40.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sahil_sadhwani_0d966c0592/i-replaced-gitpython-with-zlib-and-struct-and-gits-own-file-format-fought-back-4aea"
  },
  "original_language": "en",
  "account": "A weekend-long project revealed the challenges of reading a Git repository's object database directly, without using GitPython or similar libraries. The scanner was designed to search for API keys and credentials within a repository's history, even in deleted files. The scanner's development was hindered by Git's on-disk format, which included two specific issues: a variable-length integer encoding that appeared identical but wasn't, and a repository size large enough to demonstrate the scanner's limitations.\n\nReading a loose Git object proved to be straightforward, with just three lines of code. The object's ID was the SHA-1 of the decompressed buffer, including the header. This allowed for free verification of every read: an assert statement compared the SHA-1 of the raw data against the expected SHA-1. However, after running Git's garbage collection (git gc), all loose objects were consolidated into a single .pack file, rendering the scanner ineffective on repositories cloned from GitHub.\n\nA .pack file is composed of a sequence of objects, each with a variable-length header containing its type and size, followed by zlib-compressed content. Among the seven object types, two were not purely content; they were instructions. OFS_DELTA and REF_DELTA objects represented instructions that copied bytes from another object or inserted literal bytes. These instructions helped Git store changes efficiently, but they posed a challenge for the scanner.\n\nThe scanner's development faced an unexpected hurdle when Git's garbage collection (git gc) was executed, resulting in all loose objects being consolidated into a .pack file. This change made it difficult for the scanner to accurately read and process the repository's contents.",
  "summary": "I spent a weekend building a secret scanner that reads a git repository's object database directly. No GitPython, no pygit2, no shelling out to git. Standard library only, because that was the rule of the hackathon I built it for. The tool itself is straightforward: find API keys and credentials anywhere in a repo's history, including in files that were deleted years ago. The interesting part was…",
  "key_points": [
    "Scanner developed to find API keys in Git repository history",
    "Git's variable-length integer encoding and .pack file format posed challenges",
    "Git garbage collection consolidated loose objects into .pack file, hindering scanner"
  ],
  "editors_take": "Git's file format complexities, particularly its .pack file structure and variable-length integer encoding, limit the effectiveness of a scanner designed to search for sensitive data within a repository's history.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}