When Your Homelab Grows Up: How SQLite Took Down My k3s Control Plane
Originally published at wostal.eu . TL;DR : My Hetzner k3s lab quietly became a platform. Dozens of operators with leader-election leases hammered the default datastore — SQLite via kine — until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores. I stopped the bleeding by truncating the WAL, then migrated the…
In his Hetzner k3s lab, the author's control plane began to struggle as it transformed from a personal setup into a production-like platform. The default SQLite datastore, accessed via kine, became overwhelmed by a growing number of leader-election leases. These leases, each updated every 2 seconds by various operators, accumulated dead revisions that led to a massive SQLite database (7.5 GB) and its associated WAL file (13.8 GB), both of which failed to checkpoint.
This resulted in high CPU usage (99%), a load average of 79 on 8 cores, and timeouts when querying the datastore. To resolve the issue, the author stopped the k3s service, truncated the WAL file, and restarted k3s. The fix proved successful, reducing the load average to 5. However, the temporary spike that followed the restart was due to a reconciliation storm as every controller and ArgoCD rebuilt their caches with full LISTs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
