SIMD-accelerated computer vision on a $2 microcontroller

Engineers have created an accelerated FAST feature detector for the ESP32-S3 microcontroller, taking advantage of its 128-bit SIMD instructions. This implementation processes a QVGA frame roughly twice as fast as the original code, finishing in around 6ms. The ESP32-S3 boasts a dual-core CPU running at up to 240MHz, featuring Wi-Fi and Bluetooth Low Energy radios for its size.

The developers discovered the chip's support for SIMD instructions and decided to experiment with them, ultimately writing a basic register allocator called "basm." Intrigued by computer vision, they wrote a SIMD-accelerated FAST corner pre-test using this advanced instruction set. The fast pixel comparison relies on a function that samples sixteen pixels at once, with each vector register storing sixteen eight-bit values.

The code then calculates the extrema count of these pixels, but they encountered a limitation: the ESP32-S3's ISA doesn't allow for direct misaligned reads. After some trial and error, they discovered an instruction that could extract a slice of two concatenated registers into another one, using it to read and extract the east and west blocks.

This allowed them to process the pixel data, despite the chip only supporting signed eight-bit comparisons. They transformed the range of the values from [0, 255] to [-128, 127] by XORing each value with 0x80, enabling the arithmetic and comparisons to work correctly. With these optimizations, the developers improved the throughput of the FAST feature detector by nearly 220%, reaching 11.2MP/s in their testing.

This performance is ideal for real-time computer vision tasks, allowing the ESP32-S3 to handle a 30fps VGA stream for a cost of just $2.