I shipped a redaction library, then found it leaking a private key

A zero-dependency library called llm-stream-guardrails was developed to mask secrets and PII from an LLM's output stream. However, it was discovered that the library had a bug, which allowed for the leaking of a private key. The library held a PEM private key within a PEM block, and the mechanism that held the key trusted the footer. If the footer was mislabeled, the key was read as closed and only the header was masked. The body of the key streamed through in cleartext.

The tests for the library were designed to assert that a well-formed key block was masked. However, they did not assert the absence of a leak on a malformed block, as the tests were written for inputs that the developers thought of. This allowed for the bug to go unnoticed.

To fix the bug, the library's tests were changed to stop testing examples and start testing properties adversarially. This meant that for any input, the library would check if a value that the policy should mask ever reached the output. The library was then fuzzed to test roughly 320,000 chunkings, and a leaked body no longer reached the output.

The four issues that led to the leaking of the private key have been fixed in version 0.7.9 of the library. The regression tests now ensure that the streamed-equals-whole invariant holds across these fuzzed chunkings. A public advisory was also published for the issue: GHSA-777v-7mvx-mxgp. The main lesson from this experience is that a passing test only proves that the code does what you asked, but it says nothing about what you did not. For a redactor or any system whose job is to withhold, a missing test can be the whole product.