5 Ways to Build Retrieval Architecture for Candidate Search (and Stop Duplication)

To build a retrieval architecture for candidate search and prevent duplication, use staged retrieval with explicit collections, bounded queries, and source context that remains consistent throughout retries. Deduplication should be part of the retrieval quality rather than a post-processing job.

Operational trade-offs depend on whether your team operates Elasticsearch or a managed vector index like Pinecone or Weaviate. Infrai is a lightweight option that provides a thin HTTP layer for vector calls, allowing any HTTP client to interact with the vector surface without needing an SDK.

The retrieval architecture should preserve a specific contract, including the candidate document identifier, tenant, access-control metadata, source URL or internal document ID, chunk text, and a score. Raw documents should be stored separately, and a stable sourceKey should be derived from tenant, document ID, and content version. New parses of the same version should overwrite existing chunks, while changed versions should create new sets and retire the old ones in a controlled job.

A deterministic key is crucial to prevent duplicate ingestion after worker timeouts, as multiple chunks may be committed before the connection is lost. Idempotency is mandatory, given that standard queues are at-least-once. A chunk object includes tenantId, documentId, version, ordinal, and text, and a sourceKey function generates a stable key for deduplication.

When interacting with Infrai, use a bounded query wrapper to manage operational policies like using the real vector query route, bearer authentication, exponential backoff for rate limit responses (429), and hard failures for other non-success responses. The query is read-only, so the retry key should be a trace key rather than a write idempotency token. Use the same deterministic sourceKey across upsert attempts to ensure updates rather than adding duplicate candidates.