Kubernetes Has Garbage Collection. Why Did We Build Another One?

Kubernetes already contains built-in mechanisms for managing resource lifecycles, such as owner references, cascading deletion, Job TTLs, CronJob history limits, and finalizers. These features are effective when dealing with resources that have a clear owner relationship or a single lifecycle rule. However, when cleanup transitions from being a property of one resource to becoming a policy that applies across different resource classes, built-in mechanisms may fall short.

In environments with diverse temporary resources, such as test Jobs, generated ConfigMaps, short-lived credentials, transient namespaces, evidence artifacts, and dead-letter records, each resource may have unique disposal requirements. A Job might be disposable after one hour, while its associated evidence should remain intact. Similarly, a generated ConfigMap can be safely deleted after a run, but a ConfigMap carrying customer configuration may be owned by another controller.

This led to the development of Zen Cleaner, a garbage collector designed to evaluate explicit cleanup policies across allowed resource classes while preserving protection and ownership boundaries. The primary concern should be ensuring that the collector never deletes critical resources, such as evidence, dead-letter data, or component-owned durable state, which should remain under the control of their respective owners.

Finalizers, although powerful, are not the ideal solution for this scenario. They allow an owning controller to block deletion until it has completed a required action but do not naturally express global policies, such as collecting Jobs after 2 hours or transient ConfigMaps after 6 hours. Instead, a higher-level garbage collector controller can be built to interpret these policies using Kubernetes APIs.

Instead of resorting to shell scripts or custom deletion commands, Zen Cleaner provides a more reliable and maintainable solution. It allows for policy-based resource deletion, incorporating features like selectors, per-kind TTLs, retries, reason codes, rate limiting, and dry-run capabilities. By encoding these policies within the controller, it becomes easier to test, reason about, and ensure the safe and predictable deletion of resources.

Ultimately, the rule of thumb is to utilize Kubernetes-native lifecycle mechanisms whenever they are sufficient. Reserve the use of a policy controller for situations where lifecycle spans unrelated resource kinds, different classes require different TTLs, explicit protection is necessary, ownership must be checked before deletion, or deletion decisions require durable or auditable reason codes. In such cases, the added complexity of a controller is justified.