Three holes in a green CI, and a lock for the free plan
What this is about. On a site we worked on recently, the test pipeline went green across the board, and we looked into what green actually covered. Three things turned out not to be covered at all. This is how we found and closed each one, on a free GitHub plan. Why. A green CI tells you that the tests it ran passed. It doesn't tell you which tests it didn't run, or whether anything waits for it…
The test pipeline of a project the reporter worked on was green, indicating all tests passed. However, the reporter discovered three issues that were not covered by the green CI status, all of which were on a free GitHub plan.
Hole 1: Tests not run by CI. The backend consisted of several small services, each with its own job in CI. The reporter found that some services lacked tests and jobs altogether. These were added to the workflow file, revealing that tests were failing or crashing due to missing dependencies or unreachable URLs. To fix these issues, the reporter created a conftest.py file with a rule to skip tests that require a database when it is not set up.
Moreover, the reporter ensured that tests would display an error instead of silently skipping when a database was unavailable in CI.
Hole 2: Tests building different schemas than production. The service created its tables by running migrations when the container started. However, the tests built the tables directly from the models using create_all(), leading to mismatches between the test database and production. To address this, the reporter added a test that ran the real migrations on an empty database and compared the result with the models using Alembic.
This test found schema drift on the first run and ensured the database remained unchanged, even though the tests had passed.
Hole 3: The system lacked a wait mechanism for green CI status. The repository was private and on a free plan, making GitHub branch protection unavailable. Although a commit could go red in CI, it could still be deployed shortly after. To prevent this, the reporter implemented a small script that checked if the commit being deployed matched the one on the main branch, found the CI run for that commit, and waited for it to complete successfully.
This check also handled special cases such as docs-only commits and overrides for when CI was slow. The script was tested against a fake GitHub CLI in a throwaway repository using various scenarios, including green, red, cancelled, and untested code.
Overall, the reporter's findings and fixes resulted in a more reliable CI system that covered all aspects of the project. By adding tests to CI, addressing schema mismatches, and implementing a wait mechanism for green CI status, the reporter ensured that the pipeline would effectively inform developers of potential issues before deployment.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
