Bulk YouTube transcript extraction for AI pipelines: what breaks at scale

Extracting transcripts from YouTube videos in bulk for AI pipelines can be a challenging task. One popular approach is to use a bulk extractor, but this often proves to be unreliable at scale. The author of this post recounts their own experience building a custom pipeline to address the issues they encountered with the bulk extractors.

Some of the key challenges the author faced include rate limits kicking in, duplicate videos appearing in multiple playlists, and output formats drifting. They found that treating bulk work as a loop over single-video calls was not a scalable solution, as it provided no job state, checkpointing, or a clean way to retry failed videos without rerunning all the preceding ones.

To overcome these issues, the author recommends making the batch the primary unit of work. Each video should be treated as an item within a job that is tracked from start to finish. This approach allows for individual videos to succeed, fail, or retry on their own, providing a more robust and reliable workflow. The author suggests picking playlists as the input when they exist, as they are already a coherent dataset, and snapshotting the video list at submit time to avoid issues with playlists changing during the run. For cases where exact video IDs are known, URL lists can be used instead.

Deduplication is an essential step in the process. Video-level deduplication (same ID appearing twice) should be done during the extraction phase to ensure only one transcript is generated. Near-duplicate detection (re-uploads, mirrors) requires text similarity and becomes important once your corpus reaches a certain size. The author suggests preferring manual tracks or scoring quality for large corpora instead of chasing perfection per video.

The author emphasizes the importance of exporting transcripts in JSON format, as it provides timestamps and metadata that can be easily aligned with audio, sliced into time windows, and chunked at natural pause boundaries. CSV is also useful for analysts and warehouse loads, while TXT fits fine-tuning runs that require bare text. JSON should be considered the master format, as it can be quickly derived into other formats.

To handle rate limits imposed by YouTube, the author recommends pacing every request with a delay of 1.5 seconds between requests, never sending two requests simultaneously. They also suggest backoff on HTTP 429 responses, logging per-video errors instead of aborting the batch, and checkpointing so that a dead run can be resumed instead of starting over from scratch.

The author concludes that for training RAG models, manual captions should be preferred where available, while auto captions can be used at scale with a quality filter applied afterward. Videos without captions are expected failures, and it is recommended to log the IDs, finish the batch, and route the leftovers to an ASR system separately if necessary. For vector stores, JSON format is recommended, as it contains metadata that maps directly onto document fields.