From Hand to Throttle: Building a Gesture- Controlled Drone the Right Way

This article explores the process of building a gesture-controlled drone, from detecting hand landmarks to translating those gestures into commands for the aircraft. The process involves three stages: detecting hand landmarks, classifying hand gestures, and translating those gestures into drone commands.

Human pose estimation is the process of detecting and tracking the positions of body parts in images or video. In the context of gesture-controlled drones, this involves identifying key landmarks on the hand, such as joints and fingertips, using computer vision techniques. MediaPipe, an open-source framework for building real-time ML pipelines, provides a hand model that tracks 21 keypoints per hand, which is sufficient to distinguish various hand positions.

Stage 1 of the process involves detecting these landmarks using MediaPipe. A simple Python script can be used to detect hand landmarks in real-time using a webcam. The script installs the MediaPipe library, initializes the hands solution, and reads frames from the webcam. For each frame, the script flips the image to simulate left/right hand control, converts it to RGB, and processes it with the hands solution. If hand landmarks are detected, they are drawn on the frame, and the frame is displayed.

Stage 2 involves classifying hand gestures based on the detected landmarks. The idea is to translate the positions of the landmarks into discrete commands for the drone, such as take off, land, forward, or up. Two approaches can be used for gesture classification: rule-based finger states or a small neural network. Rule-based finger states determine whether each finger is extended or curled by comparing the fingertip position to the joint below it.

A small multilayer perceptron can also be trained on a few hundred labeled examples per gesture for more complex gesture recognition. To ensure reliable gesture recognition, a debounce mechanism is implemented using a gesture buffer that only triggers a command when a gesture is consistently recognized over several frames.

Stage 3 focuses on translating the recognized gestures into drone commands. The DJI Tello drone is used as the example platform due to its rich Python API. The gesture classification output is used to send commands to the drone, such as take off, land, or move forward. The integration of gesture control into the drone's control system allows for intuitive and natural control of the aircraft, opening up new possibilities for drone applications.