Get YouTube transcripts for every new upload from the channels you follow (for RAG, summaries and research)

YouTube transcripts offer valuable text for AI tasks, including conference talks, earnings-call recaps, tutorials, and interviews. However, reliably obtaining them at scale can be challenging. While YouTube's official Data API allows downloading captions only for owned videos, public videos provide captions that are visible under the "Show transcript" option.

The popular open-source Python library, "youtube-transcript-api," efficiently handles small jobs, but at scale, issues like IP blocks from cloud servers and keeping track of processed videos arise. To address these concerns, a step-by-step approach can be employed to obtain YouTube transcripts for every new upload from the channels you follow, specifically for use in Retrieval Augmented Generation (RAG), summaries, and research.

The first step involves using the "YouTube Transcript Scraper Actor" on Apify. This tool accepts URLs of videos in various formats, such as watch, youtu.be, Shorts, and live streams. It retrieves transcriptText, timestamped segments (start, duration, text), language, whether captions were auto-generated, word count, and video metadata (title, channel, views, duration, publish date, tags). The tool also includes features to retry and rotate proxies when YouTube rate-limits, ensuring a smooth operation.

Next, to monitor channels and only pay for new videos, set up a configuration with the channels you follow, such as "@somechannel" and "@anotherchannel". Schedule the process daily, allowing it to check the latest uploads (up to 15 per channel from the public feed) and deliver only videos that haven't been delivered before by this monitor. This setup ensures that money is only spent on new data, with a cost of $0.008 per transcript ($8 per 1,000). Videos that are already delivered, unavailable, or lack captions are free.

Once the transcripts are obtained, they can be seamlessly fed into RAG or a summary generation tool. It's crucial to chunk the transcripts on segment boundaries to maintain timestamps, enabling precise citations of specific moments within the videos. This approach ensures that your summary digest only includes new videos, eliminating the need for "have I processed this one?" bookkeeping and preventing re-embedding of duplicates into your vector store.

In case caption corrections are needed, the "New + changed" mode comes in handy. It re-delivers a video only when its captions have changed, indicated by the transcriptChanged: true flag. This feature helps maintain the accuracy of your research and summaries.

However, there are honest limitations to this approach. Videos without captions will not generate a transcript, and the tool does not perform speech-to-text conversion. The channel monitoring covers the latest 15 uploads per run, allowing the archive to grow over time. Older video URLs can also be pasted directly for individual processing.

Respecting YouTube's terms and copyright is essential. Transcripts should be used for summaries, research, and accessibility purposes, but creators' content should not be wholesale republished. To try this setup, visit the "youtube-transcript-scraper" on the Apify Store. The system is designed to only pay for new data, ensuring efficient and cost-effective use of resources.

By following this approach, you can build a comprehensive archive of YouTube transcripts for your AI projects, maximizing the utility of this valuable resource while adhering to YouTube's guidelines and copyright policies.