Our integration tests skipped themselves on every Dependabot pull request

In April, our checkout system began failing for customers who paid by bank transfer. The root cause was a minor update to our payment provider's SDK, which had accidentally renamed a field in the response we used to obtain the transfer reference. Our integration tests were supposed to call the provider's sandbox and read that specific field, but after the update, they failed.

We had Dependabot set up to automatically open pull requests for dependency updates, and a workflow was enabled to auto-merge patch and minor versions once the necessary checks passed. The integration tests require a sandbox API key, which is stored as an Actions secret. Years ago, a developer had implemented a pytest skipif statement to skip the tests when the key was missing, allowing developers without access to still run the suite on their local machines.

GitHub runs workflows triggered by Dependabot as if the pull request originated from a fork. The GITHUB_TOKEN is read-only, and Actions secrets are not passed along with the run. Dependabot has its own secret store for this very reason, and we had not previously added any secrets to it. Consequently, on every Dependabot pull request, the sandbox API key was empty, causing all forty-two integration tests to report "skipped." The pytest suite then exited with a zero status, and the check was marked as green.

This situation had persisted for eighteen months, leading us to merge every dependency update based solely on the results of the unit tests. Nobody noticed the issue because skipped tests appear in grey on a page that is rarely visited when the green check mark is present. We quickly reverted the SDK to its previous version, and transfers were restored to working order within an hour. We also updated the sandbox API key to be a Dependabot secret to prevent this issue from recurring.

To catch this problem moving forward, we modified the integration job to include a final step that failed the job if more than two tests were skipped. Additionally, the auto-merge process now requires the job summary to be reviewed, rather than just checking the green status of the pull request. This change aims to ensure that tests that intentionally skip themselves, indicating their results are not valid for a particular run, are properly accounted for.