Turning Video into Line Art in the Browser with WebGL 2 and Sobel

Depth maps and line art capture different aspects of a video. Depth maps describe the spatial relationships, while line art highlights shape, composition, and visible pose changes. After integrating depth-video conversion into Timeline Studio, an open-source browser video editor, a line-art asset generator was added. This version runs locally using WebGL 2 and Sobel edge extraction, without downloading a neural line-art model.

The image-processing pipeline connects to video editing features, including previews, trimmed clips, speed curves, source audio, encoding, and cancellation. The pipeline consists of two GPU passes: first, converting the image or decoded video frame to grayscale and smoothing it; second, applying Sobel edge extraction and thresholding to create the line art. Intermediate pixels remain on the GPU rather than being read back into JavaScript between passes to reduce setup work.

The conversion process involves converting the image to luminance, smoothing it with a 3x3 kernel, and then extracting Sobel gradients to estimate horizontal and vertical edges. The output pixel calculates the edge strength based on the gradients' magnitude. The inspector allows adjusting details, line weight, and inversion, with default settings being 55% detail, 1x line weight, and black lines on white.

The preview and asset generation share the same renderer, reducing differences in thresholds, sampling, and inversion between preview and generated assets. The output dimensions preserve aspect ratio, cap the longest edge at 1080 pixels, and are normalized to even numbers. The generator outputs video at 24 fps, and the final frame duration is limited to the remaining clip duration.