Building FoxyInvoice — Chapter 7: CI/CD — push to main and it's live
This series is written in the open, from a real production system. This chapter is the deploy pipeline end to end, plus three real incidents where it quietly lied to us. [All chapters and diagrams live in the public repo.] There is no staging environment in this system. There is main , and main is production. That sounds reckless until you see the gates — and the alternative for a solo operator…
Deploying FoxyInvoice: CI/CD Pipeline and Real-Life Incidents
This chapter covers the end-to-end deploy pipeline for FoxyInvoice, including three incidents where the pipeline failed. The system has no staging environment; instead, main is production, and the gates ensure safe push-to-deploy.
The deploy pipeline runs gates first, in parallel. Unit tests, integration tests with Testcontainers, and conformance checks run simultaneously. A concurrency group serializes deploys, ensuring pushes minutes apart never interfere. SSH deploys the code, git reset --hard on the host, and BuildKit secrets are mounted during the build.
Health gates, SPA rebuild, and smoke tests follow. Each API container must report healthy before proceeding, or the pipeline fails loudly. Database migrations apply automatically on container startup. The SPA rebuilds on the host, and both health endpoints must return 200. Finally, IndexNow is pinged.
Three incidents highlight the pipeline's flaws:
1. A build failure caused unexpected success for eight hours.
2. A cancelled deploy run locked out the concurrency group, causing queueing issues.
3. A YAML label mismatch created jobs that queued indefinitely.
The pipeline's lessons include aborting failed builds, recognizing distinct failure modes, and the importance of precise label specifications. The doctrine of "forward-fix" prevents rollbacks, as migrations are irreversible. Push-to-deploy has psychological benefits for solo developers, encouraging frequent shipping and immediate feedback.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
