Node.js Catalogue Intake — Supplier Image Metadata and Hash Identity Before Processing

In a catalogue import process, treat a supplier image as content rather than a filename before processing it. Obtain its metadata, calculate a content hash, and incorporate dimensions into the identity check before sending unseen images to the smart-crop batch. Keep track of the skipped count alongside the import result, which often exceeds expectations and indicates whether the pipeline is conserving storage and cache capacity or inadvertently increasing it.

Initially, treat the filename as insignificant context, but it becomes crucial once it functions as the cache key. This incident serves as a lesson rather than a vendor roundup. The scenario involves an edtech catalogue importing course artwork and generating various aspect ratios for catalogue cards, search results, and lesson pages.

The core trade-off is between storage and cache cost versus the minimal CPU and metadata required to identify duplicates beforehand. Processing the same image repeatedly is still redundant data, a repeated object, cache entry, and downstream batch item. The fundamental principle for deduplicating supplier images using metadata and a hash in Node.js is straightforward: filenames are labels, while bytes are identity.

Metadata aids in dismissing obviously dissimilar files and provides an audit trail; a content hash detects the same bytes under a different supplier name; dimensions help maintain the comparison useful when the application needs to differentiate source geometry before smart-cropping. The definitive identity record consists of: content hash + width + height.

The hash does not assert that two visually identical JPEGs with varying encoding settings are identical files. It addresses the frequent resend case: the same file, renamed. If a supplier re-encodes an image, the bytes may change even if a human cannot discern any difference. This boundary is essential to establish before transforming this check into a perceptual-image search system.

For each candidate, record four aspects: the supplier filename, the content hash, the dimensions, and the decision. The decision is either queued or skipped, and it should also include the import identifier. This information will simplify a postmortem analysis, enabling answers to how many files arrived, how many were new, and how many were intentionally omitted from processing.

The gate described here is intentionally unexciting: the remote image service is not instructed to process an object until this code has determined that the object is new. The seen map represents a permanent lookup in production, not a temporary map that disappears when a worker restarts.