Why Benchmarking On-Device AI Turned Into Its Own Engineering Project

Benchmarking a language model on a laptop is a straightforward task that can be accomplished with just a few lines of Python code. The model is loaded, the dataset is iterated over, and results are logged to a CSV file. The tooling for this type of benchmarking is mature and well-established, and it is rarely discussed in detail because there isn't much new to say.

However, benchmarking the same model on an Android phone presents a completely different challenge. When Redacto, an on-device PII redaction app, needed to benchmark their redaction pipeline on a Samsung Galaxy S25 Ultra, they quickly discovered that standard evaluation patterns did not apply. The memory requirements of the model made it impossible to run multiple instances simultaneously, as the Android system would kill the process if it exceeded the available memory.

To overcome this limitation, Redacto had to get creative. They discovered that Android's BroadcastReceiver mechanism could be used to trigger benchmarks without creating a separate engine instance. The benchmark runner shared the already-initialized engine from the running app, allowing them to measure the performance of their app-specific task on the phone.

This required a lot of architectural adjustments and the development of a custom test harness based on ADB (Android Debug Bridge), as there was no headless mode available for benchmarking.

The Android environment introduces numerous architectural barriers that make benchmarking fundamentally different from the laptop world. Memory constraints, system UI, background services, and the Compose UI stack all compete for resources, making it difficult to isolate the model's performance. Shared memory and the presence of multiple processes further complicate the benchmarking process.

In summary, while benchmarking on a laptop is a well-understood and solved problem, moving to Android introduces a new engineering discipline that requires addressing architectural limitations and developing custom testing frameworks. The benchmarking gap between laptops and Android phones highlights the need for tools and approaches specifically tailored to on-device AI evaluation.