Five bugs in my LLM app that never threw an error

In the span of two and a half weeks, a developer created an agent capable of reading project code review history, remembering decided conventions, and storing them in CockroachDB behind a Lambda. However, multiple issues arose during the process, causing the system to behave unexpectedly.

Firstly, the agent refused to act when it lacked information due to a misunderstanding of a simple boolean condition: bool(false) was being treated as True. To address this, a Pydantic model was implemented to ensure proper validation, thus preventing the refusal from being converted into an answer due to a stray quotation mark.

Secondly, a subsequent issue caused the system to discard entire responses six times out of six. This occurred when a maintainer commented without providing a reason, resulting in a null value being returned, which Pydantic rejected as non-string data. Consequently, the entire response was discarded and replaced with a default, leading to incorrect handling of legitimate maintainer corrections.

Thirdly, the agent stored 68 out of 79 path rules as non-matching patterns due to a mismatch between SQL LIKE wildcards and fnmatch wildcards. This oversight resulted in the agent running solely on the two rules that matched correctly, covering only 63% of the repository. The fix was a single line of code to properly clean and format the patterns.

Fourthly, the similarity scoring algorithm used to match changes to existing rules was found to be ineffective. The model seemed to match any description of a change, regardless of relevance. Upon closer inspection, it was discovered that the distances between the nearest and tenth nearest rules were nearly identical for all phrasings, indicating that the similarity measure lacked meaningful signal. Consequently, the system now relies on path matching instead of relying solely on similarity scores.

Lastly, the filtering of comments to include only those from current maintainers was flawed. GitHub's author_association field indicates current access rights, not historical involvement. As a result, all comments from a former maintainer who had written over 74,077 comments on pandas from 2012 to 2025 were incorrectly labeled as CONTRIBUTOR. This filter discarded a significant portion of the data, including valuable insights from experienced reviewers who had since left the organization, without any errors or warnings.