"Your file never leaves your device": how to test that claim (and the Chromium catch)

Many web tools now claim that their operations run entirely within the browser, with no files uploaded. One such tool (Drible, featuring 53 file tools of which 48 operate client-side) discovered that the claim could be tested. This post explains how to verify such claims, including a specific issue found in Chromium that causes an otherwise-failing test to pass when it shouldn't.

The test involves running a real job in a browser, capturing every network request made during the process, and checking for any that could have transmitted files. Two checks are used: whether the request body contains a file signature (e.g., PDFs start with %PDF, JPEGs with FF D8 FF, PNGs with 89 50 4E 47, HEIC files contain ftypheic) and whether the request method is anything other than a plain GET (POST or PUT could be used to send data). The only request allowed without a visible body is an analytics beacon, provided it remains small.

The test was implemented in Playwright, with a function `watchRequestBodies` monitoring all request bodies. If a request body contains file signatures or is a suspicious non-GET request, it is recorded. A job to test the merge PDF operation was created, which expected no uploads beyond the analytics beacon.

However, a catch was discovered: in Chromium, the test would pass even when files were being uploaded, due to the way Chromium handles file uploads. When a page sends a file as a Blob or File, Chromium does not expose the body through the DevTools protocol used by Playwright, resulting in `postDataBuffer()` returning an empty buffer. This means the test could not detect the uploaded file. The privacy test would incorrectly pass, allowing file uploads to go unnoticed.

To address this, the test was modified to treat any non-GET request as suspicious, regardless of the visible body. This change ensured that the detector would catch uploads even in Chromium, preventing false negatives in privacy tests. In Firefox, the control test (uploading a file via Ghostscript) passed, confirming that the detector worked correctly. In Chromium, the control test failed, highlighting the need for this additional check.

In summary, running a file upload test in the browser requires checking both file signatures in request bodies and non-GET requests to accurately detect file uploads, with special consideration for Chromium's handling of file uploads.