How Tailscale helped find the SQLite WAL-Reset bug

In late 2023, the reliability of Tailscale's service was compromised by a persistent bug within the SQLite database. This issue led to frequent outages over a six-month period, with 19 separate instances of database corruption occurring. Due to the nature of the control plane, which only manages configuration data, the corrupted databases contained metadata about tailnets and devices, but no sensitive information such as private encryption keys or network traffic.

When corruption happened, engineers had to stop the control plane process on the affected shard, resulting in significant downtime for affected tailnets. The bug was challenging to identify, as it didn't follow any predictable pattern or trigger condition. Despite multiple attempts to pinpoint the issue, the team ultimately relied on gathering live diagnostics in production to detect the corruption.

This investigative approach proved necessary, as the corruption didn't occur on a regular schedule and could span hours to weeks apart. The incident eroded trust among users, as many saw global incident notices that didn't affect them, even though most shards and tailnets remained unaffected.