Zero-code PII sanitization for Fluentd and Fluent Bit in Kubernetes

Fluentd and Fluent Bit are popular logging tools used in Kubernetes environments. They help collect log data from applications running within the cluster. However, these tools can inadvertently collect sensitive information, such as email addresses, API keys, and card numbers, which can then be shipped to various destinations without proper protection.

To address this issue, developers traditionally implement regex filters or Lua scripts to detect and redact these sensitive fields, but these methods can become brittle and ineffective as new secret formats appear. The Zero-Code PII Sanitization approach involves using a dedicated sidecar container, called PII-Shield, to handle the sanitization process before the logs reach Fluentd or Fluent Bit.

This approach uses entropy-based secret detection, which does not rely on regex patterns, and is executed in a low-resource Go sidecar environment. The application's logs are redirected to a shared volume, and the PII-Shield sidecar tail the file to extract and redact sensitive information. The clean logs are then passed on to Fluentd or Fluent Bit for routing, without the need for any changes to their configurations.

This method simplifies the process of protecting sensitive data, eliminates the need for maintaining regex patterns, and reduces the overhead associated with parsing and scrubbing logs.