Painting with Gaussians

In January of the previous year, the author developed a tool to transform images into pixel art by reshaping a grid to follow image edges. This edge-aware pixelation algorithm proved effective in retaining edges and preserving image details. The author later discovered that the same edge information could be utilized in digital painting.

Digital painting requires the determination of brush strokes' placement, size, and direction. Edges in a painting can be seen as contours around objects, while their absence signifies flat areas where broad strokes are sufficient. The author set out to create a digital painting program that uses brush strokes derived from image structure, allowing users to interactively modify the painting through sliders.

The project also served as an opportunity to test the Jolt library for building non-trivial applications. In this post, the author will explain how the project came together, the ideas that worked, the ones that didn't, and whether the final result resembles a painting.

The first step is understanding what a brush stroke is in computational terms. A brush stroke in oil or acrylic painting is an elongated mark with a center of color that fades toward its edges. Its orientation results from dragging the brush across the canvas. The stroke is translucent at the edges and overlaps with other strokes, allowing painters to lay down broad color blocks first and then add finer details with smaller, more translucent marks.

A 2D Gaussian splat is found to map well onto this concept. It has a mean, representing where the stroke lands, a covariance matrix representing its stretch and rotation, and color and opacity. The covariance can encode brush direction and elongation, with the major axis pointing along the stroke and the minor axis across it.

Rendering a field of splats using standard over-compositing, where each splat occludes what's behind it with its alpha, produces a result similar to a natural painting model. This approach allows marks to layer and blend together, although it doesn't offer the same fidelity as actual paint. The effect is closer to digital painting using tools like GIMP or Krita.

Several implementations of this idea already exist, such as DrawingWithGaussians and 2d-gaussian-splatting-Art, but both use a gradient descent approach. This method seeds random splats and iteratively adjusts their positions, shapes, and colors to resemble the target image. This approach is slow, opaque, and results in a lossy reconstruction of the input image rather than a painting-like output.

Since the author already had a solution for extracting edge information from images, they didn't see the need to evolve the image blindly. Instead, they used the extracted edges to guide the painting process. The edges provide information about detail density and stroke orientation without relying on gradient descent.

The author began by following the reference rasterizer, which uses additive blending. The pixel equation is: pixel = background + Σ(intensity × color). This method works well in the fitting regime, allowing the optimizer to learn colors that compensate for overlap. However, directly seeding thousands of splats based on pixel colors and rendering them additively creates excessive overlap. In some pixels, the sum reached 22.06, resulting in pure white blobs.

To address this issue, the author utilized the standard over-operator from alpha compositing. This approach ensures each splat occludes what's behind it by its alpha, preventing the summed color from exceeding 1.0. This method also cleanly separates color sampled from the image and opacity. The author used the same source photo throughout the process, recording progress as new ideas were added to demonstrate what each concept contributes to the final painting output.