How I Built an In-Place Android Screen Translator with Jetpack Compose, ML Kit, and Gemini Vision

Android screen translation tools often require users to manually capture screenshots or deal with intrusive overlays. ALST, an open-source real-time Android screen translator, solves this by translating any screen content in-place without leaving the active app. The system architecture follows Clean Architecture and MVVM/MVI design patterns, partitioned into single-responsibility modules for core capture, OCR, translation, overlay, data storage, UI, and system integration components.

ALST uses a Single-Pass Vision AI approach with Google Gemini 3.6 Flash Multimodal Vision API for single-step text extraction, context understanding, and bounding box coordinates. A Dual-Engine System enables either Cloud AI Mode (Gemini 3.6 Flash) or 100% offline On-Device Engine (ML Kit Text Recognition v2 + On-Device Translation) based on user preference. Memory management is optimized for Android 14+ by recycling bitmaps immediately after processing to prevent OutOfMemoryError crashes.

In-Place Overlay renders translated text directly over original text using a TYPE_APPLICATION_OVERLAY WindowManager view, scaled coordinates, and Jetpack Compose Canvas. Deep system integration allows ALST to operate via a Foreground Service, draggable FAB button, Quick Settings Tile, and handles permissions securely. Privacy is prioritized through a BYOK model, storing no telemetry or user data. The open-source project is hosted on GitHub and available for download on the release page.