Wiring Android's WorkManager to a Quantized On-Device LLM for Background Summarization

This story explains how to integrate a quantized large language model (LLM) with Android WorkManager to perform background summarization reliably. It outlines a pattern for wiring WorkManager to llama.cpp via JNI to handle chunked document summarization without causing memory issues, Doze-mode deferrals, or a frozen UI.

The article recommends considering the memory constraints on mid-range Android devices when selecting a model tier. Models like Phi-2 require less than 2GB RAM, while larger models exceed available RAM on 6GB devices. It is advised to target sub-1B INT4 models or sub-1.5B INT4 models for background Workers without foreground promotion.

The tutorial provides guidance on setting constraints, chaining Workers for chunked documents, and promoting to a foreground service when necessary. It emphasizes the importance of setting `setRequiresBatteryNotLow` for inference workloads and setting `setExpedited` for user-triggered summarization to avoid Doze-mode deferrals.

Implementing a foreground service is crucial when using a 3B model, as it provides up to 3 minutes of guaranteed execution time. The article also highlights the importance of avoiding silent OOM kills, profiling peak RSS on minimum-spec devices, and not loading the entire document into a single Worker.

In conclusion, the story emphasizes that three key decisions determine the success of this architecture: profiling peak RSS to select the appropriate model tier, using chained Workers for documents over 1,500 tokens, and setting `setExpedited` for user-initiated work to avoid Doze-mode deferrals.