I Built an Adaptive Hand Cricket Game Using React, MediaPipe, and TensorFlow.js

The author wanted to create an engaging game, not a typical CRUD application or chatbot. They developed Adaptive Hand Cricket, a web-based game where players use their webcam to play hand cricket. The game recognizes hand gestures in real-time, and as the player scores, the computer becomes increasingly difficult by employing randomness, live gesture reading, and machine learning.

The game uses React and Vite for the frontend, MediaPipe for real-time hand landmark detection, TensorFlow.js for learning player patterns, and Webcam input through getUserMedia. LocalStorage is used to remember previous batting behavior. Supported gestures include various finger and thumb combinations, with each gesture corresponding to a specific number. The game loop involves the computer committing to a move, the countdown, revealing both moves, and calculating the score or wicket.

The computer initially used completely random moves, which became monotonous. To make the opponent more engaging, the author added three decision strategies - RANDOM, ML PEEK, and Peek. The ML PEEK strategy allows the computer to inspect the player's gesture at a random point during the countdown if it gets a valid reading. The learning player patterns strategy uses TensorFlow.js to analyze the player's last 5 moves, overall frequency of numbers 1-9, and recent frequency of numbers 1-9, producing 63 input features. The network is intentionally small, consisting of three dense layers and a 9-way Softmax output.

The machine learning aspect was not the most challenging part; rather, it was dealing with thumb detection across different hand orientations, synchronizing the countdown and capture timing, preventing invalid camera frames from affecting gameplay, and managing game transitions cleanly. Despite being a small game, it touched on various topics, including computer vision, React state management, probability, feature engineering, browser ML, and real-time interaction design.

The author suggests adding a short multi-frame voting window for more accurate predictions, implementing proper ML-vs-random prediction accuracy tracking, creating player profiles, comparing neural networks with simpler approaches like Markov chains, and improving gesture classification for rotated hands.