Running Production With Zero Stored Logs
I run a Kubernetes platform without a centralized log archive. When I say that, people often imagine something else: short-term retention, logs with sensitive information removed, or something similar. Let me explain it in simpler terms. This platform has no centralized mechanism for collecting, forwarding or archiving application logs. We’ve The post Running Production With Zero Stored Logs…
This Kubernetes platform runs without a centralized log archive. The system comprises over 15 microservices using K3s, with nodes located on VPS servers in different areas. The nodes communicate over a private WireGuard network. A constraint was set during design: observability cannot be used to collect user information. Consequently, logs are not collected or archived centrally.
Instead of logs, nodes send metrics. Prometheus ingests these metrics and applies label restrictions. If metrics contain unauthorized labels, they are removed before storage. Metrics are protected by strict configurations, ensuring user data isn't retained. Application instrumentation is designed to avoid exposing sensitive user information in metrics.
Alerts are based on user-visible issues and calculated from metric series, so logs aren't necessary for alerts. While many worry about debugging issues without logs, the high-resolution series allows issues to be pinpointed quickly. If needed, one service can be switched to verbose mode to diagnose problems.
However, this system has its drawbacks. Diagnosing issues requires discipline, as excessive verbose logging can accumulate unwanted data. Teams must be cautious about information retention and ensure temporary diagnostics remain temporary.
Written by urgent.news from Container Journal's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
