My CI hadn't run a single one of those tests in two months and stayed green the whole time

The source material discusses an issue where a continuous integration (CI) system reported a suite of tests as passing for two months, even though the tests were missing a crucial dependency. The dependency was not explicitly declared in the development environment, but it was already installed locally. As a result, the tests using this dependency would skip instead of failing, leading to a misleading green checkmark.

The author added a guard in the form of a pytest sessionfinish hook, which raised an exit status if the number of collected tests fell below a minimum threshold. This guard helped identify when tests had stopped running unexpectedly. The author also suggested adding the -rs option to pytest, which prints the reason for every skipped test, making it easier to identify the cause of the issue.

To fix the problem, the author added the missing dependency to the project's optional dependencies list. They also removed the importorskip line for the dependency, as it was a misrepresentation of the requirement. By properly declaring the dependency, the CI system would no longer skip the tests and would report the failure correctly.

The key takeaway from the story is that a seemingly green CI build can hide underlying issues, such as missing dependencies. Properly declaring and testing dependencies is essential to ensure the reliability and trustworthiness of the CI system. The author emphasizes the importance of using the pytest --collect-only option with the -rs flag to identify skipped tests and understand the reasons behind them.