Cloudflare K2: serverless event streams

Cloudflare has introduced Cloudflare K2, a serverless event streaming service aimed at addressing the challenges of traditional Remote Procedure Call (RPC) architectures. K2 allows producers and consumers to decouple, storing events as an ordered log on the Developer Platform. This enables consumers to read events at their own pace, even if producers send large volumes of data or if consumers become unavailable.

K2 is implemented as a partitioned, durable log on top of R2 object storage, allowing it to scale to vast quantities of data and retain information over long periods. It works by accumulating writes in-memory on an edge service and then writing them as segment files, which overcome the cost of writing and reading each one. Ordering and strictly incrementing offsets are achieved using R2's atomic operations without requiring a separate coordination service.

K2 differs from Queues and Basin Pipelines, which are designed for different use cases. Queues are suited for tracking individual items of expensive or time-consuming work, while K2 is designed for high-scale data movement, long-term retention, and fan-out consumption. Basin Pipelines, on the other hand, is a serverless ingestion service for sending events to object storage or Iceberg tables.

To create a K2 stream, you can use the CLI (cf), Wrangler, the dashboard, or API. Once a stream is created, events can be produced to it for processing and storage.