Automating immersive reading

Storyteller, a complex system designed for automated reading, has evolved over time. Initially, it was a simple Python script that took an ebook and an audiobook as input and produced a new ebook file with audio synchronization metadata. However, the scope of Storyteller has grown significantly, now encompassing a full-stack web application, native apps for Android and iOS, KOReader plugins, and upcoming apps for macOS, watchOS, and tvOS.

The core of Storyteller's functionality lies in its alignment algorithm, which automatically aligns an ebook with an audiobook, identifying where each word in the ebook is spoken in the audiobook. This is achieved without any manual input from the user, using the EPUB specification's Media Overlays system to embed the audio and synchronization information into the EPUB.

The result is an immersive reading experience where the reader app highlights each sentence as it is narrated by the narrator.