{
  "id": 20037929,
  "title": "I got a RAG app running on a 512 MB free server. The trick was not running the embedding model at all.",
  "url": "https://urgent.news/2026/11/14/i-got-a-rag-app-running-on-a-512-mb-free-server-the-trick-was-not",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-14T19:58:45.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/rrhinos/i-got-a-rag-app-running-on-a-512-mb-free-server-the-trick-was-not-running-the-embedding-model-at-1m2h"
  },
  "original_language": "en",
  "account": "Here is the story written in my own words:\n\nBuilding a RAG (retrieval-augmented generation) application that runs entirely on free services and tiers is challenging, but it can be done. The main issues are the limited resources available on free web hosting services and the limitations of free LLM APIs.\n\nThe first problem is that the embedding model used to convert text into vectors does not fit into the 512 MB of RAM and 0.1 CPU provided by many free web services. Instead of running an embedding model on your own server, use a cloud inference service like Qdrant Cloud. This service can embed text for you, so your server only needs to send plain text.\n\nTo store documents and retrieve them for answering questions, use Qdrant's free cluster with the cloud inference feature enabled. The sentence-transformers/all-minilm-l6-v2 model is listed as free for use. The app's Flask server only needs to send text to Qdrant, while Qdrant handles the embedding.\n\nOne challenge is that LangChain's QdrantVectorStore expects a local embeddings object, which goes against the goal of using free services. A custom LangChain retriever was created to pass raw text to Qdrant's query_points method instead.\n\nA second issue is that Groq, a popular free LLM API, shut down its llama-3.1-8b-instant model for free and developer-tier accounts. The recommended replacement, openai/gpt-oss-20b, is a reasoning model that thinks before answering. However, its thinking tokens count toward the max_tokens limit, so a lower cap like 512 tokens can lead to cut-off or empty answers. To mitigate this, use a higher max_tokens limit like 1024 and a low reasoning_effort setting.\n\nAnother limitation is the token limits of Groq's free tier. Each question sent to Groq carries the retrieved chunks, so the token usage adds up quickly. Even with short documents and answers, a single question can cost about 600-700 tokens, allowing roughly 280 questions per day before hitting the daily token cap. This is sufficient for a demo or portfolio project but not for real public traffic.\n\nTo prevent the app from dying due to inactivity, a free weekly GitHub Actions job was set up to hit a deep health endpoint. This endpoint performs a cheap Qdrant read and never calls the LLM, so it can't consume the token budget. The job runs on a schedule and checks the health of the Qdrant cluster.\n\nLastly, a bug was encountered where a trailing newline in the Qdrant API key caused every Qdrant call to fail and print the full key in the logs. To fix this, every environment setting read from the settings was stripped of whitespace using the .strip() method.\n\nThe final stack includes Groq's openai/gpt-oss-20b model, which is a free reasoning LLM. By addressing these challenges, it's possible to build a RAG application that runs entirely on free services and tiers.",
  "summary": "Most \"build a RAG app\" tutorials work fine in a notebook. Then you try to put one online on a free host, and it falls over. I wanted a document Q&A app that answers from my own files, cites its sources, refuses when the answer isn't there, and runs entirely on free tiers with no credit card . Here's what broke along the way, and the setup that finally worked. Live demo: https://rag-0.onrender.com…",
  "key_points": [
    "Use cloud inference service Qdrant Cloud to embed text instead of running an embedding model locally",
    "Create custom LangChain retriever to pass raw text to Qdrant's querypoints method",
    "Mitigate Groq LLM API token limit by using higher maxtokens (1024) and low reasoningeffort"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}