Treat a crawler skill's output as a versioned interface

When sharing a saved crawler workflow, it is crucial for downstream users to be able to rely on the output structure of the saved workflow. Simply finding the same button is not enough. The downstream code needs to know what field names to expect, how missing values should be handled, and how to differentiate a complete collection from an interrupted one.

This article provides a contract for a reusable extraction skill, focusing on an application-side design example rather than specific implementation details. The contract should be created before saving the skill to prevent potential issues caused by changes in extraction paths that may inadvertently break downstream importers.

The consumer should first understand the code or person receiving the result. The key fields to identify a row, absent values, and pagination termination should be defined. For a directory collection, a source-specific record ID and source URL are important, while an email field could remain null if not visible. It is essential to decide on these rules before saving the skill to avoid unexpected consequences when the extraction path changes.

The output should be structurally valid but may only cover part of the requested data. A row limit is a deliberate stop, not evidence that the directory has ended. If pagination stops early due to a blocked page, the response should indicate a partial collection with an explicit status. When an authoritative expected set is available, perform a separate coverage check to report observed data and traversal outcomes. It is not appropriate to infer completeness based solely on the absence of an exception.

To validate the response, a small Python validator can be used. This example checks for a specific contract but does not validate the website, URL, or row values. The validator raises errors for incorrect data types, unsupported schema versions, unknown stop reasons, missing field values, duplicate record IDs, and non-list rows. It allows optional fields and an empty collection while preserving the business meaning of different scenarios. Synthetic fixtures should be tested to ensure the validator behaves as expected.

Testing the contract should occur separately from testing the crawler. Test the validator with valid responses, partial blocked responses, empty responses, duplicate IDs, missing source URLs, unsupported versions, and non-list rows. This establishes the behavior of the local check without addressing browser access, pagination accuracy, or extraction accuracy on a live portal.

Afterward, test the actual saved skill with authorized access and new inputs while separately inspecting source evidence and the returned structure. Keep these results distinct from the synthetic checks.

Changes to a contract, such as switching from a local record ID to a global identifier or modifying date formats, should be treated as contract decisions, not just field name changes. Record the skill revision and contract version with each run. When introducing a breaking change, retain a representative old result and exercise the migration before replacing the importer.

A version label is only useful when consumers actively check it. For optional values, clearly document whether null indicates not observed, unavailable, or not applicable. If these distinctions are necessary, consider adding a separate status field instead of using null to convey multiple incompatible meanings. A well-defined output contract provides a clear boundary for the next caller, even when the underlying website changes.