My tests could fail. They still could not tell me I was wrong.

A test can fail without confirming that the code is incorrect. The writer still shipped with the wrong result this week. The test being used was: assert.equal(sumOf(splits), transaction.total). This test can't catch if the sign convention is reversed, as both sides of the equation come from the code. The writer builds software to convert IIF files into QIF format for small business accounting software.

One example of the conversion: $150 split between two categories, with the IIF file showing a negative amount and the QIF file showing positive amounts. The mutation test revealed that flipping a correct sign would make the test fail, while flipping an incorrect sign would also make the test fail. This shows that the test doesn't verify whether the code is correct.

The writer builds a QIF file to test the conversion, with the goal of finding a sign error. The test fails, and the writer realizes that the bug is not related to signs but rather to how QIF handles payee and memo fields. The writer discovered that IIF carries both payee and memo fields, while QIF only has a payee field. The generator writing code assumed that description was equal to MEMO or NAME, which caused the payee field to be lost.

The bug was found after reading the whole code chain and realizing that the test is not checking everything the upstream parser produces. The writer learned the importance of testing assumptions, especially when they rely on someone else's software. Writing a test costs time, but it can provide answers that no amount of additional testing can.

In this case, the trade was worth fifteen minutes of testing, as it prevented a hidden bug from causing issues for users.