Python Retrieval Architecture: Customer-Support Knowledge Base Crawl Boundaries (and Freshness)

Retrieving customer-support knowledge from the web involves establishing crawl boundaries and managing information freshness. A well-defined crawl policy should be treated as a product policy to ensure only relevant and up-to-date information is indexed. For a property-management support bot, suitable content includes approved help articles, lease-policy explanations, and maintenance playbooks. Pages containing resident records, private tickets, or data dependent on live accounts should be excluded from the corpus.

To enforce crawl boundaries, a Python pipeline can be implemented before embeddings are created. This pipeline filters URLs, rejects disallowed source classes, and incorporates a freshness deadline into the indexed records. The pipeline can be applied to any crawler and vector index using generic interfaces. The policy should be reviewed by someone outside the search team to ensure transparency.

Key considerations for crawl boundaries include:

1. Source class: Differentiate between public help, internal playbooks, and policy documents.

2. URL: Only allow HTTPS URLs from approved hosts (e.g., help.example-property.com and ops.example-property.com).

3. Path prefixes: Restrict crawling to specific path prefixes (/articles/, /maintenance/, /policies/).

4. Blocked path parts: Exclude pages containing "/resident/", "/ticket/", or "/account/" as they may contain sensitive information.

5. Freshness deadline: Assign freshness deadlines based on source class (e.g., 30 days for public help, 7 days for internal playbooks, and 14 days for policies).

Each indexed document should carry metadata such as the canonical URL, source class, last-modified timestamp, and owner. The owner is crucial when an article contains outdated contact information. If a page cannot be identified as having an owner, it should be placed in a review queue instead of being treated as current.

By implementing these crawl boundaries and freshness management methods, a customer-support knowledge base can be maintained within strict retrieval controls while ensuring the information remains up-to-date and safe for use by the support bot.