Cloudflare K2: serverless event streams

Cloudflare has introduced Cloudflare K2, a serverless event streaming service, in public beta. This innovative solution addresses the challenge of decoupling producers and consumers in traditional Remote Procedure Call (RPC) architectures. By aligning scale and time between producers and consumers, K2 allows for independent processing of data by multiple consumers, preventing event loss when downstream services are unavailable or overwhelmed.

K2 functions as a durable event streaming primitive on Cloudflare's Developer Platform, storing events as an ordered log. Consumers can read the data in various ways, such as splitting reads across multiple consumers or delivering all messages to all consumers. The service is fully serverless and scales to handle vast quantities of data, with long-term retention ensuring that consumer downtime does not result in data loss.

At its core, K2 implements a partitioned, durable log on top of R2 object storage, enabling it to scale to enormous volumes of storage. If users are ready to begin using K2, they can create their first stream in seconds using the provided guide. Initially developed to serve as the ingestion layer for Basin Pipelines, K2 was built due to the need for a durable buffer on the edge, which is crucial for Pipelines' stream processing engine that operates on a pull-based model.

Cloudflare's global infrastructure presents unique challenges when deploying traditional distributed systems like Apache Kafka. To overcome these challenges, Cloudflare leverages R2, an object storage system, which offers extremely durable storage (11 9s!) and strongly consistent APIs. By offloading replication and consensus to the storage layer, K2 simplifies the application layer, making it cheaper and higher performing.

This approach also allows for independent scaling of compute and storage, enabling the storage of vast quantities of historical data at low cost.

To build a log on top of object storage, K2 accumulates writes in-memory on an edge service before writing complete files or segments to R2. This method ensures ordering and strictly incrementing offsets using R2's atomic operations, eliminating the need for a separate coordination service. However, this approach results in higher produce latencies due to the slower write speeds of object storage compared to local disks.

In K2's initial release, this adds approximately 1 second of produce latency at the 99th percentile of response times.