I taught my hand gestures to run an AI coding agent

A few weekends ago, the author grew frustrated with typing prompts into a terminal and sought an alternative way to control an AI coding agent. The solution involved using hand gestures to interact with the agent. The setup consisted of a webcam capturing the hand, MediaPipe tracking the landmarks, and three specific gestures mapping to three actions on an Anthropic-powered coding agent.

Pinch, spin, and two-finger movement each had specific meanings: pinch instructed the agent to write code, spinning the index finger indicated brainstorming, and moving two fingers up and down signaled the execution of the written code. Initially, the author attempted to use MediaPipe's newer Tasks API (HandLandmarker), but encountered a GPU crash on their Mac.

After spending excessive time troubleshooting, they switched to the legacy mp.solutions.hands API, which resolved the issue. The gestures proved to be more complex to implement than they initially appeared. Pinch detection was straightforward, while running fingers required tracking vertical oscillation and spinning fingers utilized the trajectory of the index fingertip.

Both gestures were managed using a rolling 1.5-second buffer of landmarks, triggering an action once per occurrence. The author intentionally decided not to sandbox the agent's code execution, as the core purpose was to demonstrate the agent writing and running code in real-time. The first test with a camera and no threshold tuning yielded successful gesture recognition.

Upon completing the full loop with a real camera, gestures, and Anthropic API calls, the author confirmed the system functioned as intended. Future enhancements included developing a small 3D avatar that would mirror the hand during recordings, providing a more visually appealing demonstration. The author sought feedback from others who have faced similar challenges with MediaPipe's Tasks API on Apple Silicon or had opinions on gesture-controlled technologies.