On-Device AI in Kotlin

This Kotlin tutorial teaches readers how to run a large language model (LLM) directly on a user's device, eliminating the need for servers or API keys. It begins with a simple chat exchange and gradually introduces advanced features such as multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling, and RAG.

Before diving into the code, each concept is thoroughly explained, making it accessible for beginners in on-device AI. Running AI on-device offers several advantages: it works offline, ensures privacy by keeping user data on the device, provides low latency, and eliminates cloud costs. However, on-device models are smaller and less powerful than frontier cloud models.

NobodyWho library is used throughout the tutorial, wrapping llama.cpp in Rust and offering bindings for Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It supports running models locally in .gguf format on Android and desktop JVM (Linux, macOS, Windows). The tutorial includes instructions on how to add the library to your build file, load a model, and communicate with it.

For Android, lifecycleScope or viewModelScope should be used instead of runBlocking. In a real chat interface, streaming tokens as they arrive is essential to avoid a blank screen until generation finishes. Multimodal models can ingest images and audio by using a multimodal LLM and its projection model, which converts images and/or audio into tokens the LLM can consume. Speech to text can be achieved by integrating Whisper models in ONNX format through SpeechToText.