Translated captions in a Chrome extension: tab audio in MV3, and one stream instead of two services

A Chrome extension allows users to display translated captions on a Meet, Zoom, or Teams call running in a browser tab. No additional bot is required; the audio originates from the tab itself. The extension was built using Manifest Version 3 (MV3), which employs a service worker. However, there were challenges in capturing tab audio, as the service worker could only access a stream ID and create an offscreen document containing an invisible page for these APIs to function.

The first issue encountered was that once the tab was captured, the sound ceased reaching the speakers, causing the user to lose access to the call. To resolve this, the extension routed the stream back to the output using an AudioContext, createdMediaStreamSource, and connected it to the destination. Another challenge was that the extension could not capture a tab with an active stream, as the stream persisted in the offscreen document even after closing it.

To address this, the extension periodically checked the status of the captured tabs until the stream became inactive.

The extension aimed to translate speech in real-time, initially using Deepgram for speech recognition and DeepL for translation. However, DeepL billed by source length, causing issues with interim results, which grew and were sent again in full. This led to increased costs. To mitigate this, the extension switched to Soniox, a provider that recognizes and translates in one stream. The Soniox server issued a temporary key with a time-to-live limit, enabling seamless handling of calls of any length.

However, the extension faced challenges in billing accuracy and final latency. The billing was calculated from the socket opening, not the first phrase, leading to inaccurate charges. Additionally, identifying silent tabs from provider quietness proved difficult, as both cases resulted in an increase in the PCM counter. The extension was still working on improving final latency and determining whether the tab was silent or if the provider was quiet.

The author welcomed feedback on the extension's MV3 capture part and acknowledged assistance from an AI assistant for translation and editing.