YouTube transcripts for RAG: fetch, chunk and embed captions in 30 lines of Python
Disclosure: I built the Apify Actor used in this tutorial. This post was written with AI assistance. A talk, a lecture series or a product channel is a knowledge base that happens to be stored as video. To search it or feed it to a language model you need the captions as text, split into pieces, each piece turned into a vector. This tutorial does that with three pieces: an Apify Actor that…
To perform Retrieval-Augmented Generation (RAG) with YouTube transcripts, three key components are required: an Apify Actor that fetches and returns timestamped transcripts, a chunking mechanism to segment the transcript into smaller pieces while preserving the start time, and a local open-source embedding model to convert the chunks into vectors for search and retrieval.
To begin, ensure that you have Python 3.11 or newer installed on your system. Additionally, you will need an API token from the Apify platform, which allows you to interact with the YouTube Transcript Scraper actor. The cost of using the actor is $0.001 per transcript delivered, equating to $1.00 for processing 1,000 videos. Note that videos without captions, private or unavailable videos, bad links, and playlist/channel listings are free—only transcripts that are returned to your dataset incur a cost.
The free Apify plan typically has ample monthly usage credits to handle several hundred videos easily.
To set up the environment, install the necessary Python packages: `apify-client`, `sentence-transformers`, and `numpy`. If PyTorch is not already installed, it will be required for the embedding model and will consume a significant portion of your disk space.
First, connect to Apify using the provided API token:
```python
import os, json
import numpy as np
from apify_client import ApifyClient
from sentence_transformers import SentenceTransformer
client = ApifyClient(os.environ['APIFY_TOKEN'])
```
Next, fetch the transcripts for the specified YouTube video links. For this example, we'll use the channel of the popular educator, 3Blue1Brown:
```python
run = client.actor('highbrow_fame/youtube-transcript-fast').call(
run_input={
'videos': ['https://www.youtube.com/@3blue1brown'],
'language': 'en',
'maxVideosPerSource': 20
}
)
```
The transcript data is returned as a structured dataset containing video information, including timestamps, segment durations, and textual content. This dataset is essential for the subsequent chunking and embedding steps. Each video record includes a list of segments, where each segment contains start time, duration, and the corresponding text.
Finally, the pipeline includes a chunking function that slices the transcript into smaller segments, each no larger than 60 seconds of speech. This ensures that the chunks are manageable for vector embedding and efficient search. The chunks are then embedded using a local SentenceTransformer model, enabling semantic search capabilities within the video content without requiring the video files themselves.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
