What our coverage number did not protect
Engrava's test suite had 3,845 tests and 94.22% line coverage. There was more test code in the repository than production code. By every number we had, the thing was well tested. Then we ran a mutation audit over it and found that the suite did not protect a broken concurrency guard, several deletion paths, five configuration sections, two of its own safety tests, or a field that would silently…
Engrava's test suite boasted an impressive 3,845 tests with 94.22% line coverage. It seemed like the code was well-covered and thoroughly tested. However, a mutation audit revealed that the suite failed to protect several critical areas. In one group, six guards that validated values were found to be unnecessary, as they discarded the validated results.
These guards could only be accessed by a caller running within the same process, posing no real risk to users. In another group, two safety tests were found to be ineffective, as they merely confirmed the presence of a guard rather than detecting actual issues. Lastly, a significant finding was discovered in the decay_multiplier field, which could cause incorrect data to be stored and later retrieved, potentially leading to data loss.
Despite these vulnerabilities, Engrava claimed their suite was solid and no advisories were needed. However, the mutation audit revealed that the suite did not provide the protection it claimed, leaving potential security gaps in their code.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
