A test said the server started. I deleted the server. It still passed.
Here is a test from a real, well run Node project: test ( ' server starts ' , async ( t ) => { const app = build () await app . listen ({ port : 0 }) t . assert . ok ( true , ' server started ' ) }) It reads fine in review. It runs green. Now delete the body of build() so the server never comes up. The test is still green, because the only thing it asserts is true . In the same file two more of…
A Node.js project demonstrates a flawed testing pattern that allows tests to pass even when the underlying code fails. In a real-world scenario, a test was executed that started a server, and despite the server never coming up due to a deleted section of the build function, the test still passed. This occurred because the test only asserted a boolean true value, rather than verifying the server's functionality.
Similar occurrences were found in multiple tests within the same project, as well as in other projects like Fastify. These patterns involve tests asserting true or other trivial assertions, catching errors that are never actually tested, and tests that run but do not verify the intended functionality. While coverage tools may indicate these tests are being executed, they do not guarantee the tests would fail if the code changes in a regression scenario.
The issue can be identified using a scanner like margyn-scan, which flags tests that assert nothing or where the assertion always holds true regardless of the code's behavior. This highlights the importance of reviewing tests thoroughly, as a green suite does not guarantee the tests would fail on a regression, and further auditing is necessary to ensure test reliability.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
