Building a read-aloud screen: phrases, tappable words and a finger that drifts

The concept of building a read-aloud screen with features like highlighted phrases, tappable words, and a "just listen" mode emerged from the desire to hear written content while following along visually. Outloud Reader is an iOS and Android app that accomplishes this by taking a photo of a printed page, or opening a PDF or pasting text, and reading it aloud with highlighted phrases. The app can also translate pages beforehand.

The reading screen proved to be the most challenging aspect of the app. The desired features included having the current phrase highlighted, the current sentence easily identifiable, and the rest of the page looking calm. Users should be able to tap any word to jump to that point in the audio, and there should be a "just listen" mode where the text is hidden. The app should also work for languages such as Japanese, Chinese, and Thai, treating them like any other language.

Initially, the developer considered word-by-word highlighting, but found it exhausting to watch as the highlight jumped around too much. They then experimented with using a UIKit text view as the base, due to its text selection and word hit-testing capabilities. However, they faced difficulties in achieving the specific visual look they wanted for the highlight.

The solution the developer ultimately settled on was to highlight phrases instead of words. A phrase consists of a few words up to the next punctuation mark, with a maximum of four words. The highlight is represented as a soft yellow plate, with the current sentence in full black and the rest of the page greyed out. Each word is its own view, allowing it to be tapped to jump to that point in the audio.

The layout of the words is handled by a custom SwiftUI Layout, allowing for precise control over line-breaking and hit-testing. To ensure smooth highlighting without creating thin white hairlines between words, neighboring plates are allowed to overlap slightly.