Schema Versioning When Your Extraction Changes

When the extraction schema changes, it is important to handle versioning carefully to avoid issues with comparing rows that were derived from different schemas or models. Store the schema version, document ID, prompt version, model ID, and extracted timestamp alongside every extraction record. This allows you to track the exact inputs and transformations applied to each data point.

There are three types of schema changes: structural (renaming fields, changing types, moving fields), additive (adding new fields), and semantic (changing the meaning of existing fields). Semantic changes are the most dangerous because they can make old data incompatible with new data without any obvious errors. To handle semantic changes, treat records on either side of the change as separate populations until you can compare them.

Additive changes can often be handled by backfilling the missing data using the raw response, which stores the model's full output. Semantic changes cannot be handled in this way and require re-extracting the affected records. Keep a record of every prompt version, as changes to prompts can also affect the output distribution. Version prompts in the same way as schema versions, and store them in version control alongside the code.

If a schema change requires re-extraction, do so in a shadow table and compare the old and new versions before cutting over. Extracted data is derived data, so each row represents the output of a function of four inputs: the document, schema, prompt, and model. Any change to one of these inputs means the rows are no longer comparable. Keeping all four identifiers with each record allows you to track changes accurately and avoid unexpected issues down the line.