Measuring How Cost Scales by Counting Instead of Timing

This story explores the concept of measuring computational complexity by counting operations instead of measuring elapsed time. Traditionally, complexity tools like PyPI's big-O estimates or npm's big-o-calculator measure the time it takes for a function to execute with growing input sizes. However, these tools can still produce imprecise results due to factors like standard error and noise.

The article introduces a new tool called countfn, which instead of measuring time, counts the number of reads, writes, and calls performed by a given function. By providing a function, a set of input sizes, and a way to generate the inputs, countfn reports how the computational work scales with the input size. The tool is implemented in both Python and JavaScript, allowing for cross-language comparisons.

The key advantages of countfn are that it provides consistent results regardless of the hardware or software environment, and it produces a clear ladder of complexity with error bars, making the results more reliable and interpretable. The tool counts three main types of operations: reads (element access), writes (element assignment), and calls (invocations of wrapped functions). Comparisons are treated separately to avoid conflating different types of operations.

The article presents several examples demonstrating the effectiveness of countfn. For instance, it shows that insertion sort has a quadratic time complexity, as the count of operations grows proportionally to n². The tool accurately measures this complexity even on different platforms, with no significant variation between Python and JavaScript.

The author also discusses the limitations of countfn. While it provides valuable insights into the computational complexity of an algorithm, it does not directly measure the actual running time of the code. Instead, it focuses on counting the fundamental operations that contribute to the overall complexity. This distinction is important because two algorithms with the same count of operations can still exhibit different performance characteristics due to factors like cache behavior.

The story concludes by highlighting the tool's robustness and its ability to catch discrepancies across different implementations and environments. By counting operations instead of measuring time, countfn provides a more reliable and consistent way to analyze the complexity of algorithms, helping developers make informed decisions when choosing or designing algorithms for their applications.