Apple's on-device model called our iPhone 17 "not eligible", so we tested everything except the model

When I first inquired about the on-device model's compatibility with a real phone, the SystemLanguageModel.default.availability method responded with .unavailable(.deviceNotEligible). This particular iPhone 17 running iOS 26.6, a relatively recent version, was purchased in mainland China where Apple Intelligence is not available.

Consequently, it became apparent that this was the primary reason for the unavailability. This posed a challenge as I was about to submit an AI-powered menu for my caption app, CapScribe, which generated keywords, titles, and emojis on-device. The only device I possess would never be able to run this feature. Ultimately, the honest account of this post is that we shipped a Foundation Models feature that we were unable to run, and the alternative testing methods we employed.

The decision-making process involved more than two options; the unavailability had more than a simple Bool status. When unavailable, a reason is provided, and different copy is necessary based on the reason. The SystemLanguageModel.default.availability can fall into three categories: available, unavailable due to device ineligibility, or unavailable due to the model being ready or not ready.

Only one of these outcomes is permanent. For instance, "not enabled" can be addressed by accessing the Settings. "Not ready" typically indicates that the model is still downloading and will resolve itself. Collapsing these scenarios into a single "AI unavailable" string allows the user with a fixable issue to understand that they're unable to use the feature, while a user with an ineligible phone learns nothing.

Initially, I considered hiding the menu entirely when the model wasn't accessible. I reasoned that a missing button would appear to be a bug to anyone who read the release notes, and a hidden menu couldn't explain itself to the user. The final implementation revealed the menu always appears, with disabled actions at the bottom, informing the user of the specific reason for unavailability.

To ensure proper testing, we designed a seam allowing the fallback path to be validated in the simulator, which serves as the default environment. The service utilizes an optional model through a one-method protocol, enabling it to either decode the JSON response or throw .unavailable(reason). When no model is injected, the service checks availability and either initiates a LanguageModelSession with a @Generable output type or throws .unavailable(reason).

The actual code path is kept concise to minimize reliance on hardware that is unavailable to me. Unit tests inject a mock, recording the prompt and validating that the input cap isn't exceeded. UI tests cannot inject any input, so there's a DEBUG-only launch argument: #if DEBUG public static func makeFakeIfNeeded() - OnDeviceAssistant ? { if ProcessInfo.processInfo.arguments.contains("-uitest-fake-assistant") { return OnDeviceAssistant(model: FakeAssistantModel()) } return nil } #endif This fake returns predetermined JSON data, consisting of a few keywords, three titles, and an emoji.

With this flag set, a UI test runs all three actions within the editor. It verifies that keyword emphasis is turned on in the style panel, that three title rows appear, and that the emoji is properly added to the caption text. Without the flag, another UI test opens the same menu in the simulator, where Foundation Models reports unavailable.

The test then asserts that the actions are disabled, and that the reason row exists and isn't empty. This second test holds the most significance for me since the unavailable path is the only accessible path on my own phone, and for many other users as well. Although the real model was not covered in this analysis, prompt quality, latency, and the reliability of @Guide(.count(3)) yielding three titles on a real device were not tested with hardware we own.

The App Review notes outline the device requirements, and real-model verification is scheduled for our roadmap, currently blocked until we acquire an eligible phone. After much deliberation, I decided to proceed with this approach due to the proportion of the codebase. The code that only runs on eligible hardware is minimal, directly interacting with Apple's framework.

Meanwhile, the code determining the user experience for everyone else runs on every build. If you're planning to integrate Foundation Models into your app, two crucial steps should be taken on day one. Firstly, assign a unique message to each unavailability reason. Secondly, establish a protocol between your feature and LanguageModelSession, ensuring that the fallback path is thoroughly tested in the simulator, which is typically the default environment.