{
  "id": 14618112,
  "title": "Our integration tests skipped themselves on every Dependabot pull request",
  "url": "https://urgent.news/2026/10/15/our-integration-tests-skipped-themselves-on-every-dependabot-pull",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-15T05:24:16.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sergey_shinder_ab2d943365/our-integration-tests-skipped-themselves-on-every-dependabot-pull-request-4a14"
  },
  "original_language": "en",
  "account": "In April, our checkout system began failing for customers who paid by bank transfer. The root cause was a minor update to our payment provider's SDK, which had accidentally renamed a field in the response we used to obtain the transfer reference. Our integration tests were supposed to call the provider's sandbox and read that specific field, but after the update, they failed.\n\nWe had Dependabot set up to automatically open pull requests for dependency updates, and a workflow was enabled to auto-merge patch and minor versions once the necessary checks passed. The integration tests require a sandbox API key, which is stored as an Actions secret. Years ago, a developer had implemented a pytest skipif statement to skip the tests when the key was missing, allowing developers without access to still run the suite on their local machines.\n\nGitHub runs workflows triggered by Dependabot as if the pull request originated from a fork. The GITHUB_TOKEN is read-only, and Actions secrets are not passed along with the run. Dependabot has its own secret store for this very reason, and we had not previously added any secrets to it. Consequently, on every Dependabot pull request, the sandbox API key was empty, causing all forty-two integration tests to report \"skipped.\" The pytest suite then exited with a zero status, and the check was marked as green.\n\nThis situation had persisted for eighteen months, leading us to merge every dependency update based solely on the results of the unit tests. Nobody noticed the issue because skipped tests appear in grey on a page that is rarely visited when the green check mark is present. We quickly reverted the SDK to its previous version, and transfers were restored to working order within an hour. We also updated the sandbox API key to be a Dependabot secret to prevent this issue from recurring.\n\nTo catch this problem moving forward, we modified the integration job to include a final step that failed the job if more than two tests were skipped. Additionally, the auto-merge process now requires the job summary to be reviewed, rather than just checking the green status of the pull request. This change aims to ensure that tests that intentionally skip themselves, indicating their results are not valid for a particular run, are properly accounted for.",
  "summary": "In April our checkout started failing for every customer who paid by bank transfer. The cause was a minor version of our payment provider's SDK, merged automatically two days earlier, which had renamed a field in the response we read to get the transfer reference. Our integration suite calls the provider's sandbox and reads exactly that field. It had passed on the pull request. Or rather, it had…",
  "key_points": [
    "Our checkout system failed for bank transfer payments in April",
    "Dependabot opened pull requests for dependency updates",
    "Integration tests skipped due to missing sandbox API key"
  ],
  "editors_take": "Automated dependency updates can create a false sense of security when silent test skips allow broken code to bypass critical integration checks during the auto-merge process.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}