Online Learning Search: Chunk Boundaries for Grounded Tutor Answers
Short answer Short answer: use structure-aware, overlapping chunks, then rerank with section and lesson metadata before an answer is generated. For an online course tutor, a chunk should carry enough context to support a citation, but stay small enough that a reranker can compare several candidates. I use the same rule for logistics search: a delivery exception needs its scan event, not an entire…
Short answer: Use structure-aware, overlapping chunks for online course tutors, but keep them small enough for reranking. A two-pass pipeline works best, splitting at semantic boundaries and attaching stable source spans. Chunk boundaries impact citation quality when a reader cannot verify a claim without reconstructing missing context.
Maintain three boundaries: source document span, instructional unit (lesson, module, or transcript turn), and retrieval text. A practical starting point is 250-400 tokens with 10-15% overlap for prose, and smaller units for tables, formulas, or quiz answers. Never split a claim from its qualifier. Parse the document into typed blocks like heading, paragraph, list, code, table, caption, and transcript turn.
Split blocks on sentence boundaries while preserving overlap. Do not duplicate entire paragraphs to appear safer in the index. Store source offsets alongside embedding text for citation purposes. When measuring performance, keep recall, citation precision, and entailment rates separate. Test the policy against real questions before tuning boundary sizes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
