Large Image Uploads: A Node.js Sharp and Multipart Retention Runbook

Managing large image uploads on a gaming platform requires careful consideration of backups, restore procedures, and thumbnail generation. To ensure tenant data integrity and prevent accidental deletion of recoverable copies, a runbook outlines a structured approach.

Key principles include:

- Deletion should be a deliberate policy decision, not an automatic operation

- Completed multipart uploads are the sole input to the Node.js Sharp thumbnail pipeline

- Originals remain private while derivatives are stored separately

- Restore access is verified through tenant authentication before serving bytes

Operational constraints dictate treating thumbnails as separate entities from originals. A comprehensive audit record should be maintained for each candidate for deletion, capturing:

- Tenant ID

- Asset ID

- Upload session

- Object key

- Retention class

- Completion time

- Restore hold status

- Policy version that enabled removal

This record provides on-call staff with a clear audit trail to explain deletions, ensuring accountability beyond mere storage provider logs. Verification of the restore process involves confirming three critical aspects:

1. The multipart session was uploaded under a valid upload ID

2. The final object is restorable

3. No thumbnail has been generated for that object

Restore verification confirms that the tenant can access the original snapshot via authorized paths, and that all derivatives remain intact. The policy must account for scenarios where:

- Snapshots are within their recovery window but derivatives have already met deletion eligibility

- Incomplete multipart sessions may be abandoned post-client retry policies, yet the original remains protected

- Abandoned multipart sessions should not automatically trigger disposal of the original object

Storage lifecycle rules should only be applied for coarse cleanup when their behavior aligns with the defined policy, ensuring the final eligibility determination is made within the application's own ledger. If the storage provider lacks the necessary audit and retention controls, it is unsuitable for this workflow. Instead, a storage boundary that can be operationally tested should be maintained. This testing should include:

- A held snapshot

- An abandoned multipart session

- A derivative whose source is still protected

The Node.js Sharp thumbnail pipeline should be designed as a separate contract from the upload coordinator and restore API. The coordinator handles multipart session management, records completion or abort events, and maintains two distinct namespaces for originals and derivatives. The thumbnail worker processes completed source keys with Sharp, generating revisioned derivative keys. The restore API authenticates tenants and snapshots before permitting private reads.

Critical to this process is ensuring that:

- Upload completion does not imply coordinator commitment

- Workers starting from part lists can handle incomplete data

- Workers starting from persisted completion transitions have a stable gate

- Transformation revisions and derivative keys are deterministic

- Originals and derivatives are stored in separate namespaces to prevent accidental deletion

- Private read URLs are short-lived and only created post-authentication

The coordinator's output remains minimal, consisting of a simple completion gate check in Go, without dependencies on storage SDKs. The key takeaway is maintaining this process as boring and straightforward as possible, focusing on deterministic outcomes and thorough verification steps.