Lessons From Building an Edge ML System for Tennis Swing Detection

For years, the author worked on constructing large-scale data systems for enterprise-level applications. They were accustomed to working with vast amounts of cleaned data in structured tables before reaching the models. However, when tasked with creating a system for real-time tennis swing detection, the author quickly realized how much of their previous expertise was ill-suited for this new challenge.

The core problem was capturing and analyzing the motion of a tennis racquet, while differentiating a swing from other movements like walking to the baseline or adjusting the grip. This required detecting the swing's onset and endpoint within a noisy stream of motion data, which constantly bombarded the sensor. The author found that the most difficult part was not the machine learning model itself, but properly segmenting the swing from the surrounding motion.

Once a swing was identified, the author needed to extract a handful of simple features from the data window around the hit, such as peak acceleration, rotation energy, swing duration, and impact sharpness. These features, combined with a lightweight decision tree classifier, allowed the system to provide real-time feedback to the player via a phone app, all within a 100-millisecond time frame. This tight latency requirement meant the model had to run directly on the device without relying on any server interaction.

The author's initial model performed well based on their own and a few friends' swings, but when tested with an actual player using a two-handed backhand, the system consistently mislabeled the swing as a forehand. This discrepancy highlighted the author's limited understanding of the dataset they were working with. The model had failed to account for the distinct rotational signature of a two-handed backhand, which was not present in the training data.

This incident underscored the importance of building a comprehensive dataset that accurately reflects the real-world variations in tennis swings. The author recognized that their labels were inherently noisy, and merely relying on a study of friends would inevitably lead to inaccurate results. They emphasized the need to gather a diverse set of labels from different players to truly test the model's robustness.

The gap between the model's performance on familiar data and unseen players proved to be a more valuable metric than the initial accuracy numbers, signaling the model's true limitations. Ultimately, the author's journey with building an edge ML system for tennis swing detection taught them the value of understanding the data pipeline, accommodating real-world noise, and prioritizing latency as a critical product attribute.