Measuring the sloppiness of code

Large language models (LLMs) have proven adept at generating code, yet this capability does not guarantee the code will be clean or efficient. While formally correct, such code can still introduce unnecessary abstractions, duplicate functionality, or make poor overall decisions. This insight is not novel; developers who have hastily coded projects have likely experienced an overflow of lines of code (LOC), leading to a loss of control and management over the codebase.

Some may argue that human oversight is unnecessary, given that AI agents can handle the voluminous code. However, even these agents struggle with identifying and rectifying sloppiness. Coming from a physics background, the author's inclination was to quantify code sloppiness through rigorous research and industry analysis. Initial expectations were high, especially after encountering positive research papers.

However, the industry's approach to the issue was largely "vibes based," with numerous optimistic statements about end-to-end coding agents and AI that can write and deploy code without human intervention.

LLMs are proficient at writing code that passes tests, but evaluating the 'sloppiness' of this code is more complex. Traditional methods, such as asking LLMs to assess code quality, are ineffective. More sophisticated approaches, like having a second model compare two code solutions, are also flawed because the model's preference can change with minor alterations to the code. While rubrics and test-writing by LLMs offer some promise, they are still far from achieving a comprehensive solution.

Human judgment remains the most reliable way to ensure code is human-readable, though this approach lacks scalability for training AI or establishing large benchmarks with multiple model providers. The simplest method for assessing sloppiness is to track changes in LOC, a measure that proved surprisingly effective in the author's research. However, this method is self-defeating if optimized for, as improving LOC would render it meaningless.

Other metrics, such as verbosity and erosion, were introduced by the SlopCodeBench research and showed promise in distinguishing legacy code from LLM-generated code. Verbosity measures duplicated and unnecessary lines, while erosion assesses how much of a codebase's mass is concentrated in a few complex functions. The author's tests revealed that code generated by AI agents is significantly more verbose and eroded compared to human-written code.

Average verbosity for agents' code was 0.33 ± 0.10, nearly double that of human code (0.15 ± 0.06), and erosion was 0.68 ± 0.20, almost double human erosion (0.31 ± 0.17).

The author further explored the evaluation process used in SlopCodeBench, which differs from standard coding benchmarks. SlopCodeBench employs an iterative approach, with models receiving multiple rounds of instructions and tests without retaining context between iterations. This design more closely resembles human coding practices but results in significant accumulation of sloppiness over time.

Even the most advanced models fail to pass strict evaluation criteria at all checkpoints, underscoring the challenge of measuring code sloppiness and the continued importance of human intuition and judgment in software evaluation.