Finding Bugs

Are randomized tests more effective than example-based unit tests at uncovering bugs? An intriguing discussion on the lobste.rs forum explores this question. One proponent of unit tests argues that generative testing should better reveal hidden flaws, so the author created a simple fuzzer that did indeed discover another bug in a specific version of the regex engine.

However, the author did not find any issues in the latest version. The author emphasizes that while they have a clear understanding of the bug they are investigating, they acknowledge that fuzzers can indeed locate it. The primary objective is to impart testing techniques rather than asserting their effectiveness. The author also stresses that creating fuzzers to discover known bugs is far from a trivial endeavor.

Despite believing in the power of generative testing, the author stresses that no test is ever fully comprehensive, and bugs will invariably be discovered elsewhere. The author cautions that whenever a test evades the fuzzer, it should be treated as a bug in the fuzzer, necessitating its improvement to detect similar issues in the future.

Only then should a fix and a unit test be implemented. The author then explains that regular expressions, being pure algorithms, are particularly amenable to generative testing. They provide a practical example, demonstrating how to generate a random string using a random number generator. The author emphasizes that while larger inputs are appealing, bugs typically arise from small yet intricate examples.

They recommend generating strings with a fixed alphabet, starting with the characters present in previous unit tests, and then randomly selecting subsets of the alphabet to create longer strings with predominantly 'a' and 'b' characters. To enhance fuzzer efficiency, the author suggests reusing memory across iterations and employing static allocation.

The author also discusses the generation of regular expressions using weights assigned to various features such as alternation, repetition, wildcards, and literals. By assigning a weight between 0 and 100 to each feature, the author can strategically select features during generation. The weights ensure that literals always have a non-zero weight and provide a mechanism to determine the distribution of selected features.

The author concludes by mentioning that compiling regular expressions can be slow, suggesting that compiling them beforehand could be beneficial.