Schema Evolution in AI Pipelines

In the realm of AI pipelines, schema evolution presents a significant challenge. Upstream systems frequently add new fields, rename existing ones, change data types, or start sending null values where they previously did not. These changes can lead to crashes in downstream consumers at inconvenient times, such as three in the morning.

A pipeline typically comprises three separate schemas: the source schema emitted by the upstream system, the internal schema, and the index schema used by the search store. Each schema evolves independently, and conflating them is the root cause of most issues in the pipeline.

The source schema is beyond your control and often changes without prior notice. The internal schema, which represents your normalized document and chunk representation, should change rarely and deliberately. The index schema, which defines the fields that the search store can filter on, requires careful consideration to avoid frequent rebuilds.

To mitigate the impact of schema changes, it is crucial to maintain a clear mapping between the source and internal schemas. This mapping acts as a shock absorber, isolating the pipeline from upstream changes. Each source should have its own adapter responsible for producing a valid internal document. If the adapter fails to produce a valid internal document, the entire batch should be quarantined, rather than the entire pipeline failing due to a single malformed document.

A key principle to adhere to is that the internal schema should remain agnostic to a specific source. Avoid including source-specific data directly in the internal schema. Instead, utilize a namespaced extras map to store source-specific data, making it available to those who need it without affecting the core internal schema.

Compatibility modes play a vital role in managing schema changes. Different streaming systems have formalized compatibility rules, which can be directly applied. Avro's schema resolution rules and schema registries provide four named compatibility modes:

1. BACKWARD: New readers can read old data. This mode allows for deleting fields and adding optional fields with default values. It is suitable when consumers upgrade before producers.

2. FORWARD: Old readers can read new data. This mode enables adding fields and deleting optional ones. It is ideal when producers upgrade before consumers.

3. FULL: Both BACKWARD and FORWARD modes. Fields can be added or removed with default values, and no other changes are permitted.

4. TRANSITIVE: Guarantees compatibility against every previous version, rather than just the last one. This mode is crucial for reprocessing historical data, as it ensures that a sequence of compatible changes does not render older versions unreadable.

When adding a new field to a live corpus, it is generally best to perform the migration gradually rather than in a single operation. This approach avoids the need for long write locks or complete rebuilds. By adding the field incrementally, you minimize the impact on serving traffic and reduce the risk of data loss or corruption.