The failures that don't fail loudly

In a week-long project, a reporter built an agent designed to upgrade dependencies and repair any broken code resulting from the upgrades. Unlike other automated tools like Dependabot, npm audit, and OSV, this agent didn't simply detect issues but actively attempted to fix them. However, the reporter discovered a series of significant bugs within the system that didn't seem to get caught by the usual detection methods.

The first issue was that the scanner was not detecting vulnerabilities in the lockfile, which contains information about resolved dependencies and their transitive dependencies. The reporter learned that to accurately detect vulnerabilities, the scanner must scan the entire lockfile, not just the package.json file.

The second problem was that the agent itself had a bug, which led it to report 124 vulnerable packages instead of the actual 3. The agent had nested the version inside the package object instead of placing it alongside the package name, causing the API to return every advisory for each package. The reporter fixed this issue with three separate subagents, each responsible for fixing one advisory. However, one of the fixes ended up getting silently overridden by another fix, resulting in the first fix being lost.

Finally, the reporter discovered that there were several ways the scanner could under-report vulnerabilities, including using aliases, pagination, link entries, truncated responses, and the gate working but its display not functioning properly. The reporter implemented changes to address these issues, such as refusing to continue if the response didn't match the query, and explicitly instructing the agent that being blocked was a correct outcome in the face of irreversible actions.