Two Kubernetes clusters for disaster recovery shouldn't cost twice as much

Most multi-cluster high availability setups operate on an active/active basis, requiring full workloads on both clusters 24/7. To avoid the unnecessary costs of maintaining a standby cluster, a user designed an active/passive multi-cluster configuration using Karmada. The system utilizes K3s to manage both clusters, with deployments, services, ingress, configmaps, and secrets propagated through a single PropagationPolicy.

Workloads are automatically routed to the primary cluster using ClusterAffinities, staying on standby until the primary becomes unavailable. In the event of a failure, Automatic failover uses a ClusterTaintPolicy to move workloads to the backup cluster without manual intervention. However, once the primary cluster recovers, Karmada does not automatically shift the workloads back.

A CronJob-based fail-back mechanism was developed to address this issue, waiting for the primary cluster to remain healthy for 15 minutes before clearing scheduler affinity, triggering a reschedule, and moving workloads back to the primary cluster. Cloudflare Load Balancer was added for traffic management, implementing primary and backup pools, HTTPS/healthz monitoring, and automatic failover, eliminating the need for an additional HAProxy layer.

This approach provides disaster recovery without the continuous expenses of running full production capacity on both clusters. A step-by-step guide on the installation, cluster join, failover hardening, fail-back script, and load balancer options is available on GitHub.