Running three AI models on one local server when your VRAM doesn't cover all of them

I attempted to run three AI models - Whisper, bge-m3, and gemma - simultaneously on my workstation. However, it immediately resulted in an out-of-memory error because the machine had enough VRAM for only one model at a time. I considered three options: acquiring more hardware, distributing the models across multiple machines, or loading them sequentially.

Since I didn't want to solve those problems yet, I opted for sequential loading. This means loading one model, using it, then unloading it before loading the next. While this approach avoids parallel processing, it adds a few seconds of load time per task. For Whisper, the load time was ~4 seconds for a 30-minute call transcription, which was unnoticeable in practice.

For bge-m3, I initially used all-MiniLM-L6-v2 for embeddings, but found it unreliable for mixed Korean/English documents. bge-m3 performed better with cross-lingual matching, albeit at a higher VRAM cost. I also evaluated gemma for image analysis tasks, comparing it with other models on screenshots and scanned documents. Gemma outperformed others in document-heavy images, but its processing time varied based on image resolution.

Despite these findings, I still need to develop a pipeline that integrates all three models into a cohesive system for processing call recordings, emails, and messages. Currently, I have working components but need to connect them into a unified output. This setup has been stable for several weeks without the initial OOM errors.