48-Hour Field Notes: My Free Server Kept Running — My Lease Expired Mid-Completion
I spent 48 hours moving a small extraction worker onto a free server and pointing it at free model access, and I expected the interesting failure to be latency. It wasn't. The interesting failure was that a slow model call and a short lease are two clocks that nobody was reconciling, so my worker kept completing work it no longer owned and my queue kept double-counting it. Have you ever watched a…
In a 48-hour experiment, a small developer moved a worker onto a free server and set it up to access free model resources. The developer expected latency issues, but the real problem was that a long-running model call combined with a short lease period caused jobs to be completed but never properly owned. This led to double-counting in the system.
The developer's setup consisted of a single SQLite file containing a job queue, a worker process that polled the queue, and each job calling a model to return structured JSON data. The developer made three assumptions that turned out to be incorrect: jobs have a state, a completed model call means the result is still needed, and inference being free means no need for a brake.
The first issue was that the worker's lease outlived the job request, causing jobs to sit in the "running" state indefinitely. The developer added a lease with an expiry timestamp and a reaper to fix this, but this introduced a second bug. The reaper was correct, but the worker was not because the "done" state was not enforced via a database column.
To fix this, the developer introduced a fencing token, which increments with each claim and must be presented with every write. If the fence does not match, the write fails and the job is removed as a zombie. The developer provided the entire SQLite-only code, including DDL, claim, complete, and heartbeat functions, as well as a test to demonstrate the issue.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
