Building Access Learn AI: Turning Lectures into Text, Summaries, Audio and Braille
A lecture is one of the most common ways we learn, and one of the least accessible. A student who is deaf can't follow the audio. A student who is blind can't read the slides. A student with a cognitive or attention difficulty may struggle with a 60-minute recording and no summary. Accessibility tools exist for each of these problems, but they are usually separate products. I wanted to see how…
Lectures are a prevalent method of learning, but they often lack accessibility. Students with deafness cannot follow audio, those who are blind cannot read slides, and students with cognitive or attention difficulties struggle with lengthy recordings that lack summaries. Various accessibility tools exist, but they are typically separate products.
The author sought to determine if a single pipeline could address all these issues, resulting in AccessLearnAI. This tool takes a lecture as input and generates a transcript, a concise summary, spoken audio, and Braille output. The article details the system's construction, challenges encountered, and potential improvements.
The concept behind AccessLearnAI is straightforward: students require different formats of the same content. Instead of developing individual features for each disability, the author created a unified pipeline that generates every format from a singular input: speech to text, summarization, text to speech, and Braille generation.
Each stage feeds the subsequent one. Whisper, an open-source speech-to-text model, was chosen for transcription due to its robust performance on varied accents and recording quality, which is crucial for classroom audio.
The system's architecture consists of three layers: the backend, frontend, and deployment. Fast API, coupled with PostgreSQL for storage, serves as the backend. React and TypeScript constitute the frontend, while Docker containers run on Google Cloud Platform (GCP), with the frontend deployed on Vercel. Fast API was selected because the machine learning (ML) code is in Python, allowing each pipeline stage to be exposed as a clean API endpoint without a translation layer between the models and the web service.
The most significant challenge arose when deploying the backend to Render's free tier, which consistently ran out of memory. This issue occurred because Whisper and T5 were loaded into memory simultaneously, surpassing the free tier's limits. This experience underscored a critical lesson: even a model that functions well locally may prove unusable in production due to resource constraints.
Managing memory becomes as vital as code management when deploying an ML system. The author's code can be found at https://github.com/vishal-mundewadi/AccessLearnAI, and a live demo is available at https://access-learn-ai-seven.vercel.app.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
