Backing Up 39 Products And Their CSVs On A 2 GB Single-Core Server
I lost an evening last March because I assumed my host's backups covered me. They didn't. The nightly snapshot on my 1 vCPU / 2 GB RAM box only kept seven days, and I needed a CSV from eleven days back. That was the Adobe 5-column upload file for a batch of 40 assets I had already submitted. Rebuilding it meant re-running make_metadata.py against a folder of files I had since renamed. I got it…
In March 2023, the author learned the importance of proper backup procedures after losing an evening due to a misunderstanding of their host's backup system. The host's nightly snapshot on a server with only one virtual CPU and two gigabytes of RAM only kept seven days of backups, while the author needed a CSV file from eleven days prior.
To recover, the author had to re-run a script against renamed files, which took considerable effort. This experience led the author to implement a more robust backup system as part of their pipeline.
The author's product catalogue consists of 39 finished products, totaling 856 MB of compressed output. This data is small enough to version control, but the raw material behind the products is not. The author divided their catalogue into three tiers: irreplaceable and small files, kilobyte-sized files, and raw assets. The small files, including metadata CSVs, upload tracker ledger, and research notes, are put into version control and synced remotely daily.
The kilobyte-sized files are synced weekly off-server, while the raw assets are regenerated rather than fully duplicated.
The author's biggest mistake was treating all files the same and not backing them up properly. Metadata files are crucial, as they contain decisions about titles, keywords, and orders. If the CSV files are lost, and the source filenames have changed, the author would have to redo editorial work, not just rerun a script. Similarly, the upload_tracker.py script is essential for ensuring safe re-runs by remembering which assets have been submitted.
On a 2 GB box, backup methods need to consider the hardware constraints. One vCPU means only one process can run at a time, so running archive jobs while a render is in progress can lead to out-of-memory errors and corrupted archives. To avoid these issues, the author measures peak RSS before scheduling backups, collects garbage after each asset, runs one file per subprocess for heavy steps, renders at the required resolution instead of high DPI, and verifies restores rather than backups monthly.
The server's file structure is organized in tiers to facilitate tiered backup and retention schedules.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
