Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug
Article URL: https://tailscale.com/blog/sqlite-wal-reset-bug Comments URL: https://news.ycombinator.com/item?id=49272832 Points: 351 # Comments: 48
In August of last year, a severe bug was discovered in SQLite, the database used by Tailscale's control plane, according to the company's investigation. The bug caused 19 separate instances of database corruption over six months, with each incident resulting in downtime for tailnets using that shard. When corruption occurred, the control plane process on that shard was stopped for database repair or restoration.
This affected devices connecting to the tailnet, as they required a list of other devices from the control plane before establishing new connections. While devices already online remained connected, they couldn't learn about changes to the network and temporarily lost access to the web-based admin console and the Tailscale API. Despite the rarity of the issue, it impacted a significant portion of Tailscale's infrastructure, leading to trust concerns.
The bug resisted initial attempts at reproduction and required the deployment of forensic telemetry to catch it.
Written by urgent.news from Hacker News Best's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
