Teaching a Computer to See: My First Week of Computer Vision

After weeks of training computers to read in natural language processing, our course shifted focus to teaching them to "see." Here are the insights I gained and what occurred when I applied real models to my own photos from Rwanda.

An image is essentially a grid of numbers, with dimensions of height, width, and three color values between 0 and 255. For example, my photo of people painting had 547 × 365 = 199,735 numbers representing various shades of red, green, and blue.

Filters are small grids of weights that move across an image, multiplying and adding pixel values. The Sobel filter, for instance, subtracts the left side from the right, producing 0 for flat areas and large numbers for edges. Convolutional neural networks (CNNs) use this principle, learning their own filters from data rather than having them pre-designed.

I tested three pretrained YOLO models on two photos: one of a painting, and another of agaseke baskets. Classification mistakenly labeled the painting as a "marimba" (39.8%), likely due to the wooden easels, and the agaseke as a "soap dispenser" (21%). Detection found four people in the painting photo when there were only three, and labeled the agaseke as "bottles" since there is no "basket" category within the model's training set. Segmentation revealed objects occupied 38% of the image area.

Next, I challenged the models further by presenting them with altered versions of the photos. Even when the image was very dark, upside down, blurry, or of low resolution, the model still identified three people, a horse, and nothing at all.

The primary takeaway is that a model only knows what it has been trained on. These models were primarily trained on photos taken outside of Africa, so they had never encountered an agaseke before. The solution lies in using local data to fine-tune a pretrained model, which is already being implemented across Africa. Organizations like Nuru detect cassava diseases using leaf photos, Smile ID verifies identities through selfies, Open Buildings maps buildings from satellite images, and BLAIVA, developed in Kigali, provides verbal descriptions of the world for the visually impaired.

In summary, an image consists of a grid of numbers (height × width × 3). Filters identify patterns, while CNNs learn these patterns. Classification, detection, and segmentation provide increasingly detailed information about the image. However, models struggle with things not present in their training data, emphasizing the importance of local data for improved performance.