Remove image backgrounds in the browser with ONNX Runtime Web (no server)

Running a background removal model entirely within a web browser is possible using ONNX Runtime Web and WebAssembly. This approach avoids the privacy concerns and hosting costs associated with sending images to a server. The project utilizes the IS-Net segmentation model, which has been exported to ONNX and quantized to int8 format, resulting in a significantly smaller file size of approximately 46 MB compared to the original 178 MB.

The runtime environment used is onnxruntime-web with the WASM backend. Additional components include plain canvas for image processing. To initialize the model, the WASM backend must be configured with specific settings such as setting the number of threads to 1, as multi-threaded WASM requires SharedArrayBuffer which necessitates special COOP/COEP headers for security reasons.

Before feeding the image into the model, it must be pre-processed. The photo is resized to a 1024×1024 canvas, then converted into a float tensor in NCHW order. This requires normalizing the image data to the 0..1 range and converting the RGB channels into a single float tensor expected by the model. The output from the model is a soft mask that needs further refinement.

Refining the mask involves normalizing the mask values to the 0..1 range, as the output range may vary depending on the input image. A guided filter is applied using the original image as the guide, which sharpens the mask edges and removes blurry halos often produced by upscaling the mask. Finally, a smoothstep curve is applied to the alpha channel for a cleaner outline.

Several challenges were encountered during development. The primary issue was the large file size of the model, exceeding cloud hosting limits. To overcome this, the model was split into smaller parts and hosted separately. Effective caching headers were implemented to improve the experience for repeat visits. Additionally, stale cache issues were addressed by appending query parameters to script tags.

The tool, accessible at picfitter.online, is free and does not require user signup. The photo remains on the user's device, ensuring privacy. However, the results may vary depending on the complexity of the background. For simple product images on plain backgrounds, the tool performs well. For more challenging cases, manual adjustments may be necessary using provided brushes. Suggestions for improving WASM inference speed and refining mask accuracy are welcome in the comments.