Yap: a particular kind of slop

Commenting code has long been a challenging task, as conveying information accurately is a difficult endeavor. It's tough to avoid making assumptions about the reader's prior knowledge, choosing the proper terminology consistently, and determining the most suitable locations for comments. Initially, it may require multiple rounds of rephrasing to produce a comment that is clear and concise.

One can start with something clumsy, only to later identify that it takes significantly longer to read than it should. Unfortunately, there is no compiler to validate the correctness of comments. A comment can be inaccurate and the code can still function properly, only revealing the issue later. Whenever there is a change in the type, function signature, or variable name, all other parts of the codebase must be updated accordingly.

Comments, on the other hand, remain unchanged even if they are incorrect from the beginning or become so over time. Generative AI models are producing vast amounts of code, and as reviewers, we are tasked with assessing a considerable portion of this code. Many individuals believe that the solution lies in employing LLMs to review the code, thereby reducing the workload on humans.

However, I am skeptical that this approach alone is sufficient to address the issue. How can we optimize the process of reviewing LLM-generated code? LLMs utilize words that we comprehend and construct sentences that are understandable. This does not necessarily imply that they are proficient communicators, as there exist various levels of communication.

Moreover, the distinction between skill levels and levels of intent should be acknowledged. For the purpose of brevity, we will simplify the discussion to just two levels. LLMs tend to generate a significant amount of "yapping," which refers to the continuous monologue they engage in to fill the context window and enhance future token inference accuracy.

Both humans and AI models exhibit this behavior when processing and understanding information internally. Yapping is a common way for humans to comprehend and process information, but not all brains are equal. If an individual frequently engages in internal monologue, they can be considered a "S-tier yapper," even if their thoughts are directed solely towards themselves.

The primary challenge is that LLMs struggle to keep their yapping confined to themselves. When reviewing LLM-generated code comments, one may encounter comments that are overly verbose and devoid of meaningful content. As an example, a 30-line comment above a straightforward type definition can be classified as "yap." Similarly, a 200-line comment above a large function with minimal commentary within the function body can also be considered "yap."

The placement of such comments can also be problematic, such as a 50-line comment at the top of a file attempting to explain something without providing clear details. By utilizing the term "yap," we can succinctly convey our concerns regarding these common issues in LLM-generated code. This shorthand allows us to identify patterns, evaluate the value provided by comments, and determine their appropriate placement.

If the AI community adopts this terminology, it may lead to more effective reviews and potentially improve the quality of future code.