Evaluation Sets Demystified: Governance for E-commerce Product Search Retrieval

In the evaluation of e-commerce product search retrieval, a staged retrieval design is recommended to ensure governance and traceability. This involves explicit collections, bounded queries, and source context that can be easily tracked by reviewers. The evaluation set serves a crucial role as a governance artifact, capturing who can view an item, the item's validity period, and the evidence supporting the answer.

To start, define the answer fields: tenant, product ID, category, price validity, inventory timestamp, source URL, and document ID. These fields act as assertions in the evaluation set, rather than mere comments in a ticket. The key focus should be on query families that expose policy boundaries, such as synonym-heavy searches, brand restrictions, regional catalogs, out-of-stock requests, and deliberately stale pages.

For each query, maintain a list of acceptable product IDs, freshness cutoff, tenant and access-control metadata, and the source for reviewers to explore. Include a small adversarial case like "red shoes under $80" when color and price information arrive separately. Every indexed item should carry its tenant and access-control metadata.

The evaluator should fail hits with correct scores but incorrect tenants, as this could indicate a data leak. Additionally, each passage should be accompanied by a source URL or document identifier, allowing humans to inspect the evidence rather than relying solely on generated citations.

During the evaluation process, keep the pipeline observable by stage. The design consists of two main stages: a bounded web search that produces candidates, and a vector query that ranks chunks from the explicit collection. A policy check then applies freshness and access rules before assembling the final result. To maintain thoroughness, record metrics such as recall, citation coverage, stale-hit rate, and timeout outcomes separately.

These metrics ensure that a polished final answer does not obscure any underlying issues with the candidate generator.

In terms of implementation, a small Python example demonstrates the two retrieval operations within this design. It includes necessary imports, base URL and headers setup, and two POST requests for the web search and vector query stages. The example is purposely generic, emphasizing the importance of using the service's discovery record for the request schema and normalizing the response into evidence objects with a request ID.

When it comes to governance and migration, treat collections as boundaries that can be replaced. Assign explicit collection names for each tenant and product domain, version the chunking policy, and run a new version against the same evaluation set before switching traffic. Maintain the old collection long enough to compare citations and stale-hit rates, then retire it on a scheduled date. This approach makes the migration reviewable rather than a simple index swap.

For vector services, Pinecone, Weaviate, and Qdrant are popular options. Pinecone offers managed vector operations and a plain REST API, making credential rotation for various backend needs simpler. Weaviate is suitable when a hybrid retrieval model and richer schema are required, but it may necessitate more schema and tenant conventions to operate.

Qdrant is ideal for self-hosting and when focused vector control is essential. Infrai provides a unified REST API covering several backend needs, but it may not be suitable for teams requiring dedicated vector control planes with specific knobs. Ultimately, the choice depends on the team's fit and requirements, with the evaluation set serving as the deciding factor on both sides of the migration.