Is Your Phone's LLM 37x Too Slow? Check This One Compiler Flag

Is Your Phone's LLM 37 Times Slower? This One Compiler Flag Can Speed It Up Part 3 of a series. The first step was discovering that llama.cpp for Android, when built with a debug Gradle variant, uses the -O0 optimization level. This results in a speed of 37 times slower than the release version. The metrics show generation at 0.5 tokens per second on debug compared to 18.5 on release, a 37x improvement.

The first token took 1863 ms on debug versus 52 ms on release, a 36x difference. Prefill with 43 tokens took 42 seconds on debug versus 0.556 seconds on release, a 75x improvement. Model load took 24 seconds on debug versus 3.9 seconds on release, a 6x improvement. The issue was easily overlooked because the disassembled binary showed the sdot instruction, suggesting all was well.

The sdot instruction only indicates the instruction set, not the optimization level. A reader on DEV pointed out that the build type is the culprit, as Gradle debug variants default to CMAKE_BUILD_TYPE=Debug. Changing the CMakeLists.txt to set the build type to Release, and telling Gradle to use Release, resolved the issue. Verification showed 82 instances of -O3 in the build.ninja file and -DNDEBUG present, while the release version had -O3 and -DNDEBUG.

The binary size dropped significantly when optimized, indicating actual optimization occurred. The optimization affected inlining, loop unrolling, auto-vectorization, register allocation, and dead code elimination, all of which are crucial for LLM inference. Without these optimizations, the code runs very slowly, even with a full CPU clock and threads on big cores.

To avoid this issue when building llama.cpp for Android, check the optimization flags and binary size. An optimized build should show -O3 flags and a noticeably smaller binary size. Optimization level should be checked first, followed by architecture flags, thread count and affinity, memory and IO, and finally, the algorithm itself.

A basic oversight like this can significantly impact performance, highlighting the importance of checking fundamental aspects before moving on to more complex factors.