{
  "id": 1403723,
  "title": "Running three AI models on one local server when your VRAM doesn't cover all of them",
  "url": "https://urgent.news/2026/08/17/running-three-ai-models-on-one-local-server-when-your-vram-doesnt",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-17T02:49:37.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g"
  },
  "original_language": "en",
  "account": "I attempted to run three AI models - Whisper, bge-m3, and gemma - simultaneously on my workstation. However, it immediately resulted in an out-of-memory error because the machine had enough VRAM for only one model at a time. I considered three options: acquiring more hardware, distributing the models across multiple machines, or loading them sequentially. Since I didn't want to solve those problems yet, I opted for sequential loading. This means loading one model, using it, then unloading it before loading the next. While this approach avoids parallel processing, it adds a few seconds of load time per task. For Whisper, the load time was ~4 seconds for a 30-minute call transcription, which was unnoticeable in practice. For bge-m3, I initially used all-MiniLM-L6-v2 for embeddings, but found it unreliable for mixed Korean/English documents. bge-m3 performed better with cross-lingual matching, albeit at a higher VRAM cost. I also evaluated gemma for image analysis tasks, comparing it with other models on screenshots and scanned documents. Gemma outperformed others in document-heavy images, but its processing time varied based on image resolution. Despite these findings, I still need to develop a pipeline that integrates all three models into a cohesive system for processing call recordings, emails, and messages. Currently, I have working components but need to connect them into a unified output. This setup has been stable for several weeks without the initial OOM errors.",
  "summary": "The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it OOM’d immediately. I’d known this was going to happen, but I tried anyway to see where the ceiling actually was. The machine is a workstation I already had, enough VRAM for any single model but not three simultaneously. I had some options: get more hardware, split across multiple machines, or figure…",
  "key_points": [
    "Attempted to run three AI models (Whisper, bge-m3, gemma) simultaneously on workstation",
    "Encountered out-of-memory error due to insufficient VRAM for all models at once",
    "Implemented sequential loading to avoid parallel processing issues"
  ],
  "editors_take": "Loading AI models sequentially rather than in parallel allows for limited hardware to handle multiple models, albeit with added load times, enabling a functional though not optimal workflow.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}