Scheduling every job on a GPU that can only hold one model
A Raspberry Pi 5 runs DNS for the network, plus dozens of other small services that don't need much CPU or memory but can't go down. A Jetson Orin Nano handles anything that needs a GPU, and its GPU and CPU share one memory pool, small enough to hold one loaded model or one active render job, never both. One flat network There's one LAN. Clients and services aren't split into VLANs or separated…
A Raspberry Pi 5 provides DNS services and runs various small applications that require minimal CPU and memory, but cannot afford to fail. The Jetson Orin Nano is responsible for handling tasks that necessitate a GPU, with the caveat that its GPU and CPU share a memory pool capable of accommodating only one loaded model or one active render job at a time.
The network is a single LAN, with no VLANs or firewalls to separate devices. This setup is chosen for simplicity and minimal maintenance, despite the risk of a compromised device potentially compromising the entire network. The Pi manages DNS and critical services, chosen for its reliability and low resource consumption, rather than performance.
When multiple tasks are sent to the Jetson simultaneously, it cannot handle them both due to memory constraints. Instead, all jobs are funneled through a durable queue, where a single gateway controls the flow of work to the Jetson, ensuring only one job is processed at a time based on priority. This prevents conflicts that could arise if two jobs attempted to load simultaneously into memory designed for a single model.
The gateway maintains order by queuing jobs according to priority, ensuring critical tasks are not delayed by less urgent ones. If a job is already running when a new one arrives, the incoming job queues behind it, rather than interrupting the current process. Although this approach adds a slight delay, restarting a job from scratch incurs a more significant loss of progress.
For rendering tasks, a separate machine equipped with an RTX 4070 Ti Super is utilized, which operates independently of the Jetson. This separation avoids any potential bottleneck or latency introduced by routing rendering work through the Jetson's gateway. Both the Jetson and the RTX box share the responsibility of handling LLM calls, necessitating queuing for this resource.
Despite the shared requirement, both machines work together efficiently, with the costs outweighed by the benefits of this configuration, which is comparable to the price of a high-performance laptop.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
