Learning fly pose from vision-language annotations
Animal pose estimation typically requires manual annotation of body parts to train neural networks for each new recording system or environmental condition. Here we use a general-purpose vision-language model (VLM) to generate annotations in multi-animal recordings of fruit flies (Drosophila melanogaster). Initial annotations follow written anatomical descriptions and are reviewed by an…
Traditional animal pose estimation demands painstaking manual labelling of body parts to train artificial neural networks for each distinct recording system or environmental setting. Researchers have now harnessed an all-encompassing vision-language model (VLM) to automatically generate annotations in complex multi-animal recordings of fruit flies (Drosophila melanogaster).
The initial annotations adhere to written anatomical descriptions and undergo review by a separate VLM examining neighboring video frames. Across 120 unseen fly images, the AI-produced labels exhibited a median deviation of 2.28 pixels from human-provided labels, achieving an 84.0% agreement rate within an eight-pixel threshold, averaged across ten distinct anatomical landmarks.
The head and eyes were scrutinized separately due to consistent differences in their placement. Review sessions bolstered agreement primarily at the feet, with the most persistent discrepancies occurring in areas with scant visual evidence. A visual inspection disclosed instances where the AI annotations proved more anatomically precise than those of a seasoned human annotator, particularly at the feet.
Utilizing SLEAP, a convolutional network trained on 300 AI-annotated fly images demonstrated comparable agreement with human labels and replicated the AI annotations on unseen images, with a median distance of 1.56 pixels. Moreover, this methodology was applied to recordings from an external laboratory, employing new AI annotations to train an independent pose network.
Despite the VLM annotation process taking mere minutes per fly, the trained networks facilitated pose estimation throughout over 700,000 frames. These findings underscore the potential of VLM annotations to expedite animal pose estimation without relying on pre-labelled training images.
Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.