Measuring the sloppiness of code

Large language models (LLMs) have become proficient at generating code, but this does not guarantee the code is clean or efficient. Excessive abstractions, duplication, and poor decision-making can still occur, despite the code being technically correct. This is not a novel realization; many developers have experienced that adding more features often leads to an increase in lines of code (LOC), making it difficult for humans to keep up with the growing codebase.

Some may argue this is not a problem, as they trust their AI agents to handle it. However, even AI agents struggle to deal with the inherent "slop" in the generated code.

Coming from a physics background, the author's initial approach to quantifying code sloppiness involved an extensive literature review and analysis of industry practices. Unfortunately, most research papers left the author disappointed, with many promoting "vibes-based" solutions. Industry hype often emphasizes the ability of LLMs to write almost perfect code and the promise of end-to-end coding agents that can generate and ship code automatically.

However, these claims are based on the scalability and verifiability of code, rather than the quality of the code itself.

Measuring the sloppiness of generated code is particularly challenging, as it relies heavily on human intuition and judgment. One approach is to let AI models judge the quality of the code, but this method is generally ineffective. Asking models to rate their own code on a scale of 1-10 is similar to a random number generator. More sophisticated methods, such as having the model compare two solutions and choose the preferred one, also fail due to the model's changing preferences when solutions are renamed.

Human judges of AI are another potential solution, but this approach lacks scalability for training AI or creating large benchmarks with multiple model providers and harnesses. A simple metric is to compare the change in the number of LOCs, but this metric becomes less meaningful if the aim is to optimize for it. Two more advanced metrics, introduced by the SlopCodeBench paper, are promising.

Verbosity measures the amount of duplicated, unnecessary, verbose lines, while erosion assesses how much of a codebase's mass is concentrated in a few large, complex functions. In evaluations, these metrics reveal that LLM-generated code is roughly twice as verbose and eroded compared to human-written code.

One reason LLMs cannot effectively address code sloppiness is the evaluation process itself. Unlike other coding benchmarks that provide the agent with a complete set of instructions at the beginning and have hidden tests to pass, SlopCodeBench creates multiple rounds of instruction and test iterations. Context is erased between checkpoints, which better mimics real-world coding scenarios where decisions accumulate over time.

Consequently, even the most advanced models fail to achieve a perfect "strict solve rate," where all tests must be passed at each checkpoint. This serves as a cautionary tale for organizations that add tens of thousands or even hundreds of thousands of LOC daily.