Online Course Retrieval Architecture: Tutor Index Economics (and Why I Chose One)

The architecture for an online course tutor retrieval system involves a staged design that includes explicit collections, bounded queries, and source context that can be traced back to the lesson and tenant. The cost of index management grows with each reprocessed transcript, and an unbounded query can lead to increased latency and token spend.

A simple approach is to use one giant index and a single similarity call, which is quick to demonstrate but difficult to explain in the context of learners receiving answers from the wrong course.

A more robust design involves separate ingestion, querying, and citation stages, each with observable inputs and outputs. The evaluation of this architecture before release should consider factors such as recall, precision, re-index cost, and citation coverage on representative lessons. This evaluation should include deliberately bad matches to test the system's performance.

The retrieval contract should specify returning up to eight chunks from the learner's enrolled course, including lesson ID and access metadata, and providing enough source text to cite the exact section.

To build a small evaluation set, real material such as coding lessons, quiz explanations, transcripts with repeated terminology, and lessons that should produce no results should be used. Additionally, adversarial pairs should be included, such as "closures in JavaScript" versus "closures in Rust." Labels should be added for the expected chunks, and recall, precision, re-index cost, and citation coverage should be measured.

Preserving tenant, course, enrollment scope, lesson, and source offsets on every indexed item is crucial. Filtering should be applied before ranking, and the same scope should be enforced again before rendering citations. An embedding version should also be stored, such as "lesson-emb-v3," to enable rollback or side-by-side recall testing after a re-index.