{
  "id": 17207858,
  "title": "Three holes in a green CI, and a lock for the free plan",
  "url": "https://urgent.news/2026/10/30/three-holes-in-a-green-ci-and-a-lock-for-the-free-plan",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-30T09:55:37.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/tdude/three-holes-in-a-green-ci-and-a-lock-for-the-free-plan-4ak0"
  },
  "original_language": "en",
  "account": "The test pipeline of a project the reporter worked on was green, indicating all tests passed. However, the reporter discovered three issues that were not covered by the green CI status, all of which were on a free GitHub plan.\n\nHole 1: Tests not run by CI. The backend consisted of several small services, each with its own job in CI. The reporter found that some services lacked tests and jobs altogether. These were added to the workflow file, revealing that tests were failing or crashing due to missing dependencies or unreachable URLs. To fix these issues, the reporter created a conftest.py file with a rule to skip tests that require a database when it is not set up. Moreover, the reporter ensured that tests would display an error instead of silently skipping when a database was unavailable in CI.\n\nHole 2: Tests building different schemas than production. The service created its tables by running migrations when the container started. However, the tests built the tables directly from the models using create_all(), leading to mismatches between the test database and production. To address this, the reporter added a test that ran the real migrations on an empty database and compared the result with the models using Alembic. This test found schema drift on the first run and ensured the database remained unchanged, even though the tests had passed.\n\nHole 3: The system lacked a wait mechanism for green CI status. The repository was private and on a free plan, making GitHub branch protection unavailable. Although a commit could go red in CI, it could still be deployed shortly after. To prevent this, the reporter implemented a small script that checked if the commit being deployed matched the one on the main branch, found the CI run for that commit, and waited for it to complete successfully. This check also handled special cases such as docs-only commits and overrides for when CI was slow. The script was tested against a fake GitHub CLI in a throwaway repository using various scenarios, including green, red, cancelled, and untested code.\n\nOverall, the reporter's findings and fixes resulted in a more reliable CI system that covered all aspects of the project. By adding tests to CI, addressing schema mismatches, and implementing a wait mechanism for green CI status, the reporter ensured that the pipeline would effectively inform developers of potential issues before deployment.",
  "summary": "What this is about. On a site we worked on recently, the test pipeline went green across the board, and we looked into what green actually covered. Three things turned out not to be covered at all. This is how we found and closed each one, on a free GitHub plan. Why. A green CI tells you that the tests it ran passed. It doesn't tell you which tests it didn't run, or whether anything waits for it…",
  "key_points": [
    "Three issues uncovered in green CI: missing tests, schema mismatches, and lack of wait mechanism.",
    "Tests added to CI to address missing tests and schema mismatches.",
    "Script implemented to wait for green CI status in private, free plan repository."
  ],
  "editors_take": "Relying on automated green status indicators without verifying test coverage or schema parity creates a false sense of security that can allow broken code to bypass deployment safeguards.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}