From video to encounter histories: individual identification and machine vision for salmonid capture-recapture monitoring

Capture-recapture models rely on individual encounter histories, but traditional fisheries monitoring typically only records counts without considering differences in detection probability. This research introduces a machine-vision system to transform long-term video footage of Atlantic salmon (Salmo salar) and sea trout (Salmo trutta) spawning runs into individual data suitable for capture-recapture analysis.

The process begins by identifying frames containing fish, saving video clips, and extracting fish-head regions of interest. Cropped images are then organized for potential re-identification.

To detect fish in the video, the binary EfficientNetB0 fish detector achieved impressive results with a validation accuracy of 99.18% and a PR AUC of 99.92%. By setting a conservative detection threshold of 0.98, the system managed to eliminate false positives while maintaining a recall rate of 94.3%. For localizing the fish-head regions within the frames, the YOLOv8n model proved effective.

Concurrently, an EfficientNetB0 ArcFace model was trained on head images of 700 individual fish. This model demonstrated remarkable accuracy in identifying known matches, achieving a 99.80% success rate among accepted pairings. Furthermore, when considering the system's performance with new, untrained identities, it exhibited an impressive image-weighted false-accept rate of just 0.81%.

Perhaps most impressive was the model's ability to generalize to previously unseen fish, achieved through zero-shot closed-set retrieval. Across a dataset of 1,416 identities that were not part of the model training process, the system still achieved an impressive 99.17% accuracy in correctly identifying these individuals as unknown.