Bug Finding Considered Harmful?

In recent discussions, the practice of finding bugs within software development has been questioned. This article explores the concept of property-based testing (PBT), a method that defines general facts about a system through blueprints or properties. By randomly sampling inputs to these properties, testers can locate bugs and help maintainers fix them.

The value proposition of testing research is to build reliable software systems, and PBT, along with other testing approaches like fuzzing and formal verification, serve this purpose with various trade-offs.

The article highlights the tension between testers and maintainers, as testers approach systems from an external perspective and lack the insider knowledge that maintainers possess. In the age of artificial intelligence, the gap between bug finding and fixing has become more pronounced. The author filed over 500 bug reports in a single month, expressing pride in their achievement but also concern for the potential delay of important software updates, such as Postgres19.

They argue that not all bugs are equally important and that prioritizing which bugs to fix is crucial.

While security bugs are essential to address, the article suggests that the value of bug finding shifts when functional correctness properties of systems are tested. Instead of simply reporting every bug found, testers should collaborate with maintainers to determine if fixing a bug is worth the effort. The author concludes that, given the asymmetry between bug identification and resolution, bug finding will continue to outpace bug fixing.

In the future, testers must carefully craft and prioritize fixes to ensure the software remains effective and user-friendly.