Cloudflare Scales D1 Control Plane to Millions of Databases
In a recent article, Cloudflare shared the redesign of the control plane of its D1 serverless SQL database to support millions of databases, avoiding centralized bottlenecks, implementing retries and idempotency, and using eventual consistency. By Renato Losio
Cloudflare has recently announced that it has redesigned the control plane of its D1 serverless SQL database to support millions of databases, addressing scalability issues and avoiding centralized bottlenecks. The new architecture is built around Postgres for transactional metadata, ClickHouse for analytics, and Workers KV for globally replicated routing data.
By keeping query processing at the edge, separate from centralized management operations, Cloudflare has implemented distributed metadata caching, connection pooling, asynchronous workflows, and periodic health checks to improve scalability and recover from failures. The new system uses 30-second heartbeats to reconcile each D1 Durable Object with Postgres, the source of truth for database existence and account limits, ensuring self-correction of failed creation or deletion operations.
Brief written by urgent.news from InfoQ's own syndicated text. Machine-written — may contain errors; check the original before relying on it.