Zero-downtime schema evolution: Auto-migrations for ClickHouse.

Auto-migrations for ClickHouse enable zero-downtime schema evolution, as demonstrated during the WClickHouse Open-Source Engineering Series. The pain points of managing schema migrations on fast-moving event streams were addressed with the development of WClickHouse TableSync.

TableSync works by comparing Pydantic model fields with the live ClickHouse table schema. When a new field is added to the Pydantic model, TableSync automatically adds the corresponding column to the ClickHouse table using the correct data type. This non-destructive approach never drops existing columns or alters existing data partitions.

The implementation utilizes the following Python libraries: typing, Pydantic, and WClickHouse. A Pydantic model class, EventModel, is defined with fields such as event_id, name, and an optional experiment_tag. TableSync then automatically adds the experiment_tag column to the EventModel table. The output confirms the successful addition of the column.

This architecture has been tested and verified against live ClickHouse server instances, with over 95% test coverage. It is built for Python versions 3.9 through 3.14 and supports Apache Arrow and Pydantic v2. For more information, visit the GitHub repository (https://github.com/wisrovi/wclickhouse) or the PyPI project page (https://pypi.org/project/wclickhouse).