Apple ML Research: Introducing Foundation Models

At Apple's Worldwide Developers Conference in 2024, the company unveiled Apple Intelligence, an advanced personal intelligence system integrated into iOS 18, iPadOS 18, and macOS Sequoia. Apple Intelligence comprises multiple highly-capable generative models, each tailored to users' everyday tasks. These models can adapt in real-time based on the user's current activity.

Two of these models are highlighted in detail: a ~3 billion parameter on-device language model and a larger server-based language model available through Private Cloud Compute. Both models are designed to perform specialized tasks efficiently, accurately, and responsibly.

The on-device model and server model are part of a broader family of generative models developed by Apple, including a coding model for Xcode and a diffusion model for visual expression in apps like Messages. Apple Intelligence is built on a foundation of privacy innovations and adheres to Responsible AI principles guiding its development.

The models are trained on Apple's AXLearn framework, an open-source project that efficiently scales training across various hardware and cloud platforms. Data used for training comes from licensed sources, public data collected by AppleBot, and user interactions filtered to remove personally identifiable information. Quality assurance measures include data extraction, deduplication, a model-based classifier, and post-training algorithms like rejection sampling fine-tuning and reinforcement learning from human feedback.

Optimizations for speed and efficiency include grouped-query-attention, shared input and output vocab embedding tables, low-bit palletization, and a mixed 2-bit and 4-bit configuration strategy using LoRA adapters. These efforts have resulted in highly capable, fast, and power-efficient models that enhance user experiences across Apple platforms.