Architectural Breakdown: We Rendered a HQ Demo Video on a Free Farm Run by Strangers
 # We Rendered a 4K Demo Video on a Free Farm Run by Strangers. Here's the Architecture That Stopped It from Melting Down at 3 AM The ticket hit at 2:17 AM on a Tuesday. Client wanted a 90‑second 4K demo video. We had a GitHub…
A 4K demo video was rendered on a free farm consisting of 37 stranger nodes, each with 8 GB of RAM and no service level agreement (SLA). The project began at 2:17 AM on a Tuesday, with a client requesting a 90-second video. As the rendering process commenced, the software encountered several critical issues that threatened the delivery of the final product.
Firstly, out-of-memory (OOM) kills occurred on edge nodes due to an unbounded per-frame texture cache without any memory semaphore. Second, event-loop stalls lasting over 5 seconds were caused by synchronous I/O blocking the asyncio event loop. Third, lock contention on output files resulted from a global threading.Lock serializing all worker writes, leading to a throughput drop to four frames per minute.
Fourth, missing frames occurred after node dropout because there was no idempotent chunk tracking or resumable uploads. Fifth, rate-limit errors from cloud storage were encountered due to unthrottled multipart uploads from dozens of parallel workers. Finally, CPU starvation was experienced on weak nodes because of a hardcoded 4-core assumption on 2-core machines.
To address these challenges, a back-pressure-driven architecture was employed, eliminating the hope-driven approach. The system consisted of four main components: Coordinator, Dispatcher, Worker Pool, and various supporting services. The Coordinator maintained a single source of truth using SQLite with Write-Ahead Logging (WAL) mode, ensuring that every chunk had a UUID, a status transition log, and a checksum.
No memory data was kept without an on-disk counterpart. If the Coordinator experienced failure, it could replay the WAL and resume from the last checkpoint.
The Dispatcher controlled traffic flow with a semaphore limiting concurrent uploads to 12, carrying out checksum verification before handing off chunks to workers. Workers communicated their heartbeat every 30 seconds, marking chunks as PENDING_RETRY if there were three missed heartbeats. A MEMORY_SEMAPHORE enforced a maximum of 22 chunks per node based on available RAM and chunk size, preventing memory spikes and explosions.
Workers were responsible for rate limiting and multipart uploading. A watchdog timer re‑assigned tasks if a chunk couldn't be completed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
