I Ran Every --selftest Block in This Repo and Found One Bug They'd All Miss
I kept seeing "Detecting Flaky Tests and Finding Their Root Causes" show up in my feed this week. I don't have flaky tests, though. I have something plainer and, it turns out, less trustworthy: six scripts in my git-manager repo that each carry a --selftest block instead of a real test framework. I run them by hand before I push, not in CI. I'd never actually asked how much they catch. So I…
I ran every selftest block in this repository and discovered one significant bug that others likely overlooked. The DEV.to API integration scripts in the git-manager repository each include a --selftest block with hand-written assertions. These scripts range from 4 to 62 assertions, covering various functions. However, despite the presence of these assertions, one critical bug remains undetected.
This bug involves the User-Agent header, which DEV.to rejects if it contains "urllib" (case-insensitive). The assertion for this issue does not exist, and mutation testing confirms that the existing tests would fail silently without this oversight. The gap exists because the selftest assertions focus on pure logic rather than side effects like network requests.
This oversight is easy to miss, as the tests only verify the function's output against fake inputs. To address this, adding an explicit assertion to check for the presence of "urllib" in the User-Agent header would ensure proper functionality. This discovery highlights the need for more comprehensive testing that considers side effects and external interactions.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.