Architectural Breakdown: Stop rebuilding from scratch: cache Docker layers on Cloud Build
 # Stop Rebuilding From Scratch: Cache Docker Layers on Cloud Build (And Actually Mean It) It was 2:17 AM on a Tuesday when I realized our CI pipeline was burning $340 a month doing nothing useful. I ran `docker history` against…
On a Tuesday at 2:17 AM, the reporter discovered that the CI pipeline was wasting $340 per month without providing any useful output. Running `docker history` revealed that the same `pip install` layer was being recreated 47 times. This 2.1 GB tarball remained unchanged for six weeks, and every build had to download it because Cloud Build workers are disposable containers that forget everything once a build finishes.
This scenario is not hypothetical; it is the default state of any team running ephemeral workers without proper cache management.
BuildKit maintains its cache in SQLite on the local filesystem. When Cloud Build spins up a worker, it creates a fresh `/var/lib/buildkit` and wipes it clean when the job is done. This means cached layers, compiled artifacts, and resolved dependency trees are discarded after each build. Using Docker Buildx's registry cache exporter solves this issue by pushing cache as OCI blobs into Artifact Registry, where cache storage costs are significantly lower.
For a typical Python project, 60% of build time is spent downloading packages, which translates to a significant portion of the CI bill going towards redundant HTTP requests.
The reporter deployed this solution across six production services, resulting in an average cache hit rate of 94% after the third deployment cycle. This represents actual, shipping builds rather than theoretical improvements. To achieve this, the reporter set specific build configurations using Docker Buildx. The Docker buildx create command was used to create a builder named `buildkit-cache` with a 6 GB memory limit and the `--use` flag to automatically bootstrap the builder.
The worker configuration in the `/etc/buildkitd.toml` file was adjusted to limit concurrent blob uploads, enable garbage collection, and set a 4 GB hard cap on total cache storage. The Cloud Build pipeline yaml included steps to bootstrap the builder, execute the build with cache-in and cache-out options, and tear down the builder to free resources.
By implementing these changes, the reporter was able to significantly improve build performance and reduce unnecessary costs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
