When Upstream Disappears: Building a Dependency Safety Net with Lockfiles, Proxy Caches and Git Mirrors
Tuần này trên Hacker News có tin repo ONNX Runtime của Microsoft biến mất khỏi GitHub. Chưa cần biết lý do là gì, câu hỏi mình đặt ra ngay là: nếu sáng mai một dependency quan trọng của team mình bị xóa, bị đổi tên hoặc bị chuyển sang private, thì CI còn build được không? Với nhiều team mình từng làm cùng, câu trả lời thật là không . Build chạy pip install hoặc npm install thẳng lên internet,…
Upstream repositories disappearing can cause major build issues. This article explains how to create a safety net using lockfiles, proxy caches, and Git mirrors to prevent such problems.
The first layer involves using lockfiles and hashes to ensure the exact dependency versions are installed. When a developer or CI runs a command like pip install or npm install, the project pulls the dependencies directly from the upstream registry, causing the entire pipeline to halt if the upstream server goes down. To avoid this, generate a lockfile with a hash that changes only if the package's contents are altered.
This way, even if the original package is removed or renamed, the installed version will match the one listed in the lockfile.
The second layer is setting up a private proxy cache for package registries. This cache stores the downloaded packages from the upstream registry, allowing the developer or CI to pull them from the local cache instead of the upstream registry. If the upstream repository removes or modifies a package, the proxy cache still contains the previous version. For npm, Verdaccio can be used to set up a private proxy cache. For Python, devpi-server can be used for the same purpose.
The third and final layer is mirroring important Git repositories. Some dependencies are not managed through package registries, such as Git submodules, Git URLs, or repositories that have been forked and modified. By creating a local mirror of these repositories, the team ensures that the exact version of the code is available, even if the upstream repository is removed or modified. The mirror can be updated on a regular schedule, and tags should not be pruned to ensure that force-pushed commits do not break the build.
By implementing these three layers, teams can significantly reduce the risk of build failures caused by upstream repository problems. The checklist provided in the article can be used to ensure that the safety net is properly set up and functioning. This approach helps to maintain a stable build process and reduces the impact of supply chain attacks on the development workflow.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.