What “local OCR” should mean in a web application

"Local OCR" is a term often used to describe an optical character recognition process that occurs within a web application without sending user data to external servers. However, determining what exactly constitutes "local OCR" can be more complex than it seems. Several factors must be considered when evaluating the privacy and security of an OCR tool running in a browser environment.

Firstly, it's essential to understand that any interaction between the user's device and external services can potentially expose sensitive data. This includes not only the OCR output but also metadata associated with the input files such as filenames, file sizes, and potentially even the contents of protected documents like PDFs. Therefore, the question shifts from merely where the JavaScript code is hosted to a more nuanced inquiry about the handling of user-derived data at various stages of the OCR process.

To address these concerns, a threat model should be established to identify what constitutes sensitive information throughout the OCR workflow. This includes the raw image data, filenames, MIME metadata, PDF passwords, intermediate rendered pages, and the recognized text itself. The guiding principle should be that none of this data should be able to leave the confines of the user's device without explicit permission.

This means avoiding the transmission of any user-derived bytes through fetch requests, form submissions, beacons, analytics payloads, browser storage, or logs.

Resources that are universally applicable, such as OCR models and character dictionaries, do not contain any user-specific data and can be downloaded without concern. However, these resources should be treated as part of the application code rather than sensitive user data. Implementing a test that specifically looks for the presence of user data in any outgoing request can help ensure compliance with this principle.

Another critical aspect is the execution of OCR algorithms. Running OCR on the main thread of a web application can lead to performance issues and a poor user experience. Therefore, it's recommended to move the OCR process to a Web Worker, which runs in parallel to the main thread. This allows for the decoding of selected image files into an ImageBitmap object, which can then be processed by the OCR worker using models like ONNX Runtime Web.

By using transferable objects, memory and CPU load can be managed more efficiently, avoiding the pitfalls of copying large image buffers back and forth between threads.

When dealing with file sizes, it's important to note that the size of the input file does not always correlate with the computational cost of OCR. A small compressed image can expand significantly once decoded, potentially exceeding the available GPU memory. Therefore, a practical pipeline should include checks for MIME types, compressed byte sizes, decoded dimensions, and a maximum safe decoding threshold. This helps prevent the consumption of excessive system resources during the OCR process.

After decoding the image, it's crucial to constrain the image to an appropriate size before feeding it into the OCR model. While a high-resolution image may contain the necessary information to accurately recognize text, it's not always required. The OCR detector can operate on a resized version of the image, which reduces the computational burden without significantly compromising accuracy.

The OCR pipeline should consist of multiple stages, beginning with detection, followed by recognition. This modular approach allows for greater flexibility and can help meet various user requirements, such as the need for higher accuracy at the cost of additional processing power.

One important consideration is separating the text detection and recognition processes. By first producing a text probability map through detection, regions of recognized text can be more accurately identified and cropped. This allows for a more efficient recognition process, as each cropped region can be processed independently by the recognition model. The output can then include not only the recognized text but also its geometric information, enabling the creation of table-oriented outputs where appropriate.

PDF documents present a unique set of challenges in a local OCR setup. Unlike standard image files, PDFs may contain page ranges, passwords, and malformed structures that can complicate the OCR process. It's essential to render only the selected pages, limit the document size and page count, and keep passwords securely in memory.

Each page should be converted to a bounded bitmap and processed through the OCR pipeline, with canvases released as soon as their OCR results are available. This ensures that resources are used efficiently and that the user experience remains responsive.

Offline support is another consideration for a local OCR tool. While the initial download of the application shell and chosen OCR models is necessary, the tool should clearly communicate the need for a network connection after the initial setup. A service worker can be used to cache immutable resources and models, but care must be taken to manage model versions separately to avoid confusion.

The UI should provide clear feedback to the user regarding the status of model loading and the capabilities of offline usage, ensuring that expectations are managed appropriately.

In addition to the main functional components, the user interface of an OCR application should be carefully translated to accommodate all user interactions related to OCR processes. This includes error messages, prompts for passwords, progress indicators, model downloads, and any other states that affect the user's experience during OCR operations. Translating these messages into the user's language ensures a seamless experience, regardless of the language they are using.

Testing the OCR application's behavior under various conditions is crucial to validate its performance and privacy claims. Tests should cover offline operation after a successful warm-up, as well as scenarios where the network is disabled. Additionally, when a user attempts to use a model that has not been cached, the application should clearly communicate the failure rather than silently switching to an alternative model. This helps maintain transparency and trust with the user.

To ensure that the privacy claim holds up under scrutiny, it's essential to have a robust set of tests in place. These tests should not only verify the functionality of the OCR tool but also its adherence to the privacy model established earlier. By focusing on the privacy claim as a critical aspect of testing, developers can ensure that their local OCR tool meets the expectations of users concerned about data privacy and security.