{
  "id": 19572453,
  "title": "Python Retrieval Architecture: Customer-Support Knowledge Base Crawl Boundaries (and Freshness)",
  "url": "https://urgent.news/2026/11/12/python-retrieval-architecture-customer-support-knowledge-base-crawl",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-12T00:08:20.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/echof76/python-retrieval-architecture-customer-support-knowledge-base-crawl-boundaries-and-freshness-294g"
  },
  "original_language": "en",
  "account": "Retrieving customer-support knowledge from the web involves establishing crawl boundaries and managing information freshness. A well-defined crawl policy should be treated as a product policy to ensure only relevant and up-to-date information is indexed. For a property-management support bot, suitable content includes approved help articles, lease-policy explanations, and maintenance playbooks. Pages containing resident records, private tickets, or data dependent on live accounts should be excluded from the corpus.\n\nTo enforce crawl boundaries, a Python pipeline can be implemented before embeddings are created. This pipeline filters URLs, rejects disallowed source classes, and incorporates a freshness deadline into the indexed records. The pipeline can be applied to any crawler and vector index using generic interfaces. The policy should be reviewed by someone outside the search team to ensure transparency.\n\nKey considerations for crawl boundaries include:\n1. Source class: Differentiate between public help, internal playbooks, and policy documents.\n2. URL: Only allow HTTPS URLs from approved hosts (e.g., help.example-property.com and ops.example-property.com).\n3. Path prefixes: Restrict crawling to specific path prefixes (/articles/, /maintenance/, /policies/).\n4. Blocked path parts: Exclude pages containing \"/resident/\", \"/ticket/\", or \"/account/\" as they may contain sensitive information.\n5. Freshness deadline: Assign freshness deadlines based on source class (e.g., 30 days for public help, 7 days for internal playbooks, and 14 days for policies).\n\nEach indexed document should carry metadata such as the canonical URL, source class, last-modified timestamp, and owner. The owner is crucial when an article contains outdated contact information. If a page cannot be identified as having an owner, it should be placed in a review queue instead of being treated as current.\n\nBy implementing these crawl boundaries and freshness management methods, a customer-support knowledge base can be maintained within strict retrieval controls while ensuring the information remains up-to-date and safe for use by the support bot.",
  "summary": "Short answer: treat crawl boundaries as a product policy, then make freshness an explicit field in retrieval. For a property-management support bot, index approved help articles, lease-policy explanations, and maintenance playbooks while excluding resident records, private tickets, and pages whose meaning depends on a live account. A small Python pipeline can enforce that boundary before…",
  "key_points": [
    "Define crawl boundaries as a product policy for relevant, up-to-date knowledge retrieval",
    "Implement Python pipeline to filter URLs, reject disallowed sources, and add freshness deadline"
  ],
  "editors_take": "Treating crawl policies as product decisions protects sensitive user data while ensuring support bots rely on verified, current information rather than outdated or private resident records.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}