One Hub to Watch Them All: Multi-Tenant Monitoring for 4 Clusters and 400+ Edge Machines with Grafana, Mimir, Loki and Alloy
"How I built a self-hosted, GitOps-managed observability hub on Kubernetes: Mimir for metrics, Loki for logs, Alloy as the only agent, tenants enforced at the gateway, and S3 as the real database." The problem Our monitoring was spread across too many places. We run a handful of on-prem Kubernetes clusters (one "hub" plus several "spoke" clusters) and more than 400 edge machines at remote sites.…
The article details the author's experience building a self-hosted, GitOps-managed observability hub running on Kubernetes. The hub centralizes monitoring for a handful of on-prem Kubernetes clusters and over 400 edge machines. The author chose to use Grafana Alloy as the only agent, Mimir for metrics storage, Loki for log storage, and S3 as the primary database.
The design is based on a hub-and-spoke model where the hub hosts Grafana Alloy, Mimir, Loki, Grafana UI, and alerting. Edge machines run Alloy locally, sending metrics and logs to the hub's API. The hub never has to interact directly with the edge sites. Object storage (S3-compatible MinIO) serves as the central database, with Mimir and Loki keeping only recent data locally (15 days for metrics, 30 days for logs).
The article covers the implementation details, including collection and metrics configuration, and log storage configuration. It also discusses the mistakes made along the way, such as manually configuring alerts and secrets management, and how they were corrected by adopting a GitOps approach and using tools like OpenBao and External Secrets Operator for handling secrets. The final configuration consists of a Kubernetes-native stack with high availability and scalability in mind.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
