I pentested my own AI hub and shipped the method, not the map

In a recent penetration test on his own AI-assisted coding infrastructure, the reporter discovered numerous serious defects within the system. The test involved a read-only audit of the code and configuration, as well as a dynamic run against the entire stack running locally in Docker. The reporter expected to find issues in the less-looked-at areas, but instead found that the majority of the defects were in controls written recently, often by the reporter themselves.

The reporter's observability hub consisted of six services within a single compose file, including a tunnel, OpenTelemetry Collector, Prometheus, Grafana, Loki, and a status API. The public surface consisted of three aggregate numbers, with everything else kept private. The reporter had implemented a privacy boundary as an allow-list rather than a deny-list, which proved to be a successful measure.

One of the significant issues identified was the lack of proper filtering of key attributes in Loki. The reporter had implemented a keep_keys function to filter keys, but the implementation was flawed, leading to sensitive information being exposed. Additionally, the reporter found that a line in the log path was meant to zero the trace ID on every record before storage, but due to a parsing error, it failed on every record, allowing sensitive data to pass through undetected.

Another issue discovered was a conditional failure in the OTTL (OpenTelemetry tracing) documentation. The reporter had written a proof that was meant to zero the trace ID on every record before storage, but due to a parsing error, it was actually failing on every record. This allowed sensitive data to be leaked through the logs.

Finally, the reporter found that a dependency PR had moved the image version for the Collector, causing the proofs to continue pulling the old image and passing false tests. The reporter fixed this issue by writing a CI gate to prevent the mistake from happening again, but also discovered that the gate itself was born broken, allowing false positives to occur.

Overall, the reporter's pentest revealed a multitude of issues within his own AI-assisted coding infrastructure, highlighting the importance of thorough testing and review of code and configurations.