The GPU that says yes and does nothing: debugging real-time hair segmentation on mid-range Android

We created a virtual mirror for hair salons using a tablet camera feed, where the customer's hair changes color in real-time as they move their head. Two constraints shaped every decision: privacy (no frame leaves the device) and hardware (salons don't buy flagship tablets). The team optimized the process by measuring frame rates on real Android devices using diagnostic tools. A Pixel 8 Pro ran at 9 fps while a Samsung A56 (Exynos 1580, Mali GPU) showed no color at all.

The issue turned out to be the MediaPipe's ImageSegmenter with GPU delegate, which ran in 5 ms and returned zero hair pixels. The fallback to CPU-only triggered on exceptions, but there were no exceptions. The team discovered that a delegate that constructed successfully might not work, and it's crucial to check the output, not just the exit code. They implemented a fallback that rebuilds the engine on CPU after five consecutive empty masks.

The team also found that the per-pixel JavaScript loop was a significant contributor to the frame budget. They moved the recolouring into a fragment shader, reducing the time from 66 ms to 0. They decoupled the mask from the draw, drawing the video on all frames using the most recent mask. This change fixed the A56, and the Pixel 8 Pro went from 9 fps to 15 fps.

They also shared the WebGL context with MediaPipe, eliminating the CPU round trip for the mask. Finally, they switched to a specialised model, reducing the APK size and improving performance.