Never create 1-row ClickHouse parts again: The built-in Buffer Manager.

Day 03 of the WClickHouse Open-Source Engineering Series reveals a solution to eliminate the need for creating 1-row ClickHouse parts. WClickHouse, an open-source library, introduces a built-in Buffer Manager that allows for high-throughput streaming writes without the need for external Redis clusters or Celery workers.

The pain points faced by microservices requiring real-time event processing include manually setting up Redis buffers, Celery flush workers, and the risk of accidentally crashing ClickHouse clusters with high-frequency single inserts. WClickHouse addresses these issues by enabling automatic buffering for high-throughput streams.

To implement WClickHouse, users simply need to import the library and initialize an instance with telemetry, database configuration, buffering enabled, and a specified buffer size. Events are then inserted into the buffer, which is stored in RAM and automatically flushed to ClickHouse when the buffer reaches its size limit. This threshold auto-flush ensures optimal performance without the need for additional infrastructure.

The benefits of this architecture include the elimination of extra infrastructure requirements such as Redis, Kafka, or queue brokers. The built-in Buffer Manager provides a zero-increment solution that reduces complexity and cost while maintaining performance. WClickHouse has been thoroughly tested and verified against live ClickHouse server instances, with 95%+ test coverage, and is compatible with Python 3.9 through 3.14, utilizing Apache Arrow and Pydantic v2.

For those interested in learning more, WClickHouse can be found on GitHub at https://github.com/wisrovi/wclickhouse and on PyPI at https://pypi.org/project/wclickhouse. The library was created by William Steve Rodríguez Villamizar, known as Wisrovi.