On-Device Body Tracking in a PWA: How AURADUEL Scores an Aura Battle Without a Server

On-Device Body Tracking in a PWA: AURADUEL's Unique Aura Battle Without a Server

AURADUEL, a game designed for players aged 16 to 24, prioritizes privacy by ensuring that all biometrics and movement analysis occur entirely on the device. No body landmarks leave the phone, eliminating the risk of data leaks. This rule guided every decision made during development, resulting in a stack composed of React 18, Vite, TypeScript, and Capacitor 8, packaged as a PWA and wrapped for iOS.

To achieve real-time AR effects, the game utilizes @mediapipe/tasks-vision 1.0.1 for gesture recognition in hands and PoseLandmarker lite (float16) for body tracking. The AR layer, implemented with pixi.js 8, displays spheres, sparks, and scores in real-time on the camera feed. Firebase provides hosting, Firestore, and Realtime Database capabilities for the mirror duel between two phones, while RevenueCat handles in-app purchases.

The WASM engine and two models weigh approximately 15 MB when gzipped, which is a constraint for viral traffic. To mitigate this, the team used third-party hosted files, including the wasm from jsDelivr and the models from Google's public mediapipe-models bucket, ensuring integrity verification against local copies. This approach reduced bandwidth costs during launch traffic.

The game's detection system runs two detectors simultaneously on a mid-range phone. However, this can strain the device, so the system alternates pose and hands detection phases depending on the game's phase. The GPU driver can fail on some devices, leading to a fallback to CPU rendering in specific situations.

Camera-related issues were encountered, such as portrait video not being square and MediaPipe returning normalized landmarks per axis. The team addressed these issues by adjusting calculations and implementing a minimum visibility and torso size visibility filter. The creativity score calculation was also refined to account for actual body movement rather than zone changes, and the energy score measures the mean speed of wrists and ankles normalized by torso height.

The Aura Score combines various factors, including presence, creativity, impact, technique, and fair play, ensuring a fair and balanced scoring system for players.