I built a full image editor that never uploads your photo — here's how the on-device AI parts actually work

The article discusses building an on-device image editor called APIC-Web, which utilizes various client-side technologies to perform image editing tasks without uploading photos to a server. The two most challenging aspects of the project were background removal and object removal.

For background removal, the article initially used MediaPipe's ImageSegmenter with outputCategoryMask: true, which provided hard 0/1 labels for background and foreground pixels. However, the resulting edges were jagged, especially around hair and shoulders. To fix this, the article suggested switching to outputConfidenceMasks: true, which returns confidence probabilities (0-1) rather than hard labels.

By applying a smoothstep contrast, erosion, and feathering techniques, the article was able to produce smoother edges using the same underlying model.

Object removal, on the other hand, does not require a trained generative model. Instead, exemplar-based texture synthesis can be used, which essentially mimics the pre-AI generation of content-aware fill. The process involves taking the painted region (masked area) plus a padded area of surrounding pixels and breaking it down into small blocks (8x8).

Each block is then processed boundary-inward, comparing surrounding texture to find the patch that minimizes squared distance (SSD) against the already-resolved parts of the target block. The matching real pixels are then copied in, preserving sharpness instead of smearing.

One crucial bug the author encountered while implementing this technique was that transparent pixels still carried leftover garbage RGB values after background removal. To fix this, the author created two separate arrays: isPainted (defining the hole) and validSrc (opaque and unpainted pixels safe to copy from).

Overall, APIC-Web proves that many image editing tasks can be handled effectively on-device using native browser technologies like Canvas, WebAssembly, and on-device ML models, providing users with privacy and control over their data.