Local Embeddings vs. API Embeddings — Why I Chose sentence-transformers
Every RAG pipeline needs to convert text into vectors. The question is where that conversion happens. You have two options: run an embedding model locally on your own hardware, or call an API that runs the model on someone else's hardware. Both work. The right choice depends on your constraints — and understanding the tradeoffs is more useful than a recommendation. This article is about why I…
Retrieving relevant information from large text corpora is a common challenge in natural language processing. The key to effective retrieval is converting text into numerical representations known as embeddings. These embeddings capture the semantic meaning of words and phrases, allowing them to be compared mathematically.
Embedding models take raw text and transform it into fixed-size vectors of floating-point numbers. In the case of the all-MiniLM-L6-v2 model used in this pipeline, each vector contains 384 dimensions. Text with similar meaning tends to produce vectors that are close together in this high-dimensional space, while unrelated text will have vectors that are far apart.
When building a Retrieval-Augmented Generation (RAG) system, you have two main options for generating these embeddings: running the model locally on your own hardware, or calling an external API that hosts the model. Each approach has its trade-offs.
Using a locally run embedding model like sentence-transformers/all-MiniLM-L6-v2 has several advantages for development purposes. It requires no additional infrastructure, no API keys, no network latency, and no associated costs. The model is downloaded once and runs entirely on the local CPU, making it fast enough for small document sets. Importantly, no sensitive data leaves the local machine, mitigating data exposure risks for documents containing confidential information.
However, local embedding models do have limitations. They have a quality ceiling compared to larger, more expensive API models. They rely on CPU resources, which can become a bottleneck for large-scale ingestion. Updating the model is cumbersome as it requires re-embedding the entire corpus. And since the model is fixed, different machines may produce slightly different embeddings, potentially causing consistency issues.
These limitations become more pronounced at production scale. For handling massive volumes of embeddings, a hosted API with GPU infrastructure is necessary. For highly accurate domains like medical or legal content, the limited quality of smaller models may not be sufficient. And in multi-user environments, a centralized embedding API ensures consistency across all services.
In this particular case, the decision was made to switch to an API-based embedding solution provided by Voyage AI. This change was made by updating a single line of code in the pipeline's configuration. The new API-based embedding function is now used for production-level embedding tasks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
