Finding "Duplicates Through Time"

The challenge of managing a vast personal photo and video library, totaling 300GB and spanning a decade, proved daunting. Space constraints forced the author to seek a solution without resorting to external cloud storage providers or purchasing additional hardware. The compression route, while tempting, risked compromising the quality of precious memories. The author settled on a more selective approach: deleting redundant large files and targeting individual photographs.

A key realization came from a friend's request for a single picture from their New York visit, only to find the author had captured seven nearly identical shots. This prompted the idea of identifying and eliminating duplicate photos taken in quick succession. The author's solution was Siegu, a local tool that finds burst-mode photographs across extensive libraries.

Siegu operates differently from traditional deduplication tools, which rely on exact byte-for-byte comparisons using MD5 or SHA256 hashes. Instead, Siegu considers both temporal proximity and visual similarity. By analyzing EXIF metadata to group photos shot within seconds or milliseconds of each other, and comparing them using perceptual hashes like pHash or dHash, Siegu efficiently identifies duplicates.

This approach reduces unnecessary CPU cycles, focusing only on candidate groups that were actually captured in short bursts. In the next installment, the author will delve into the technical aspects of Siegu's Rust implementation, EXIF metadata extraction, and the perceptual hashing algorithm employed to flag these duplicates.