Course Tutor Retrieval: Staging Context for Predictable Response Latency

In a healthtech course tutor, the primary constraint is not the vector index but the response budget. When retrieval takes 900 ms to produce usable context, a faster model cannot salvage the interaction. To address this, staged retrieval with explicit collections, bounded queries, and traceable source context is recommended. Each stage should be treated as a separate observable step, allowing for easier diagnosis of failures.

When defining the tutor's contract, start by specifying what the system should be able to answer before choosing an index. For example, a lesson on insulin titration may require the lesson section, its revision, the tenant, and the learner's access scope. The contract should be concise enough to test in a unit test.

A query includes a tenant, a course, a learner scope, and a deadline. The result should contain a stable source ID, a bounded text span, a score, and enough metadata to generate a citation. Any missing fields constitute a retrieval miss, regardless of the similarity score.

A staged design is preferred over running every backend in parallel. The first useful split is between lexical intent and semantic intent. Perform a narrow lexical pass for exact terms, then a semantic pass for paraphrases, and merge the results only after applying access filters. This staged approach reduces the need for parallel calls to multiple backends, saving time.

To verify the contract, use a replay set consisting of a current lesson, a stale revision, a document from another tenant, and a page containing the exact dosage phrase but no answer. Observe how the system behaves under different filtering conditions, such as current lessons surviving all filters, stale revisions losing to current revisions, foreign tenants disappearing before ranking, and dosage pages being retrieved for lexical intent but rejected if outside the learner's role.

When analyzing failure traces, focus on item metadata and scope loaded at the start of the request instead of tuning the embedding model. If six candidates merge into a single paragraph, inspect chunk IDs and deduplication. If citations are absent despite existing selected IDs, inspect the citation assembler.

Allocate specific latency budgets to each stage and log both elapsed time and outcome. Typical budgets include 40 ms for query normalization and ACL load, 180 ms for candidate retrieval, 120 ms for reranking and deduplication, 80 ms for citation assembly, and 60 ms for answer generation handoff. Reserve some time for network variance.

Named timeouts provide actionable insights, such as "Candidate retrieval exceeded 180 ms for collection anatomy-v3," rather than generic "Search took too long." Log counts instead of learner text to maintain privacy and focus on observable metrics.