Building a Private Classifier with Apple's On-Device Model and NLContextualEmbedding

Apple's on-device model is a small language model that runs offline on macOS and iOS devices, costing nothing per call and never sending user data anywhere. This article discusses its capabilities, limitations, and how to build a classification engine using two components: NLContextualEmbedding and the Foundation Models framework.

The model has approximately 3 billion parameters and a context window of about 4,096 tokens at launch, covering the instructions, prompt, and response together. However, the context size may change between OS versions, so it's essential to check it for your target version. Long documents won't fit, so plan to chunk them.

The model runs on the Neural Engine and GPU, making it fast enough for interactive use on Apple silicon. However, throughput varies depending on the hardware and output length, so it's crucial to measure on your own device before relying on specific numbers.

The model is well-suited for classifying and tagging text, extracting structured fields from short text, summarizing or rewriting short passages, and any use case that requires privacy or offline processing. It's not ideal for broad world knowledge, heavy reasoning or math, or long documents without chunking.

The key feature of this model is structured output. With the @Generable protocol, it can return typed Swift values instead of free text. For classification purposes, it means the model can only respond with predefined category names and confidence levels.

To build a classification engine, import the FoundationModels framework and define a struct that conforms to @Generable, specifying the allowed category names and confidence range. Create an instance of LanguageModelSession with instructions for the expected categories. Generate a Verdict struct to receive the classification results.

Before running the full language model for every item, consider using NLContextualEmbedding, a BERT-style model built into the NaturalLanguage framework. It converts text into vectors, allowing you to score items by similarity to each category without any generation. Embed each category once and cache their vectors. Then, embed the input text and compute cosine similarity against every category.

If the top score clearly outperforms the second, accept it as the classification. However, if the top two scores are close, use the LanguageModelSession to ask the model to choose between just those two candidates, using the @Generable struct for a more reliable answer. This approach is fast, uses no generation, and minimizes the risk of misclassification.