The Model Is a Variable: Compiling a Neural Network Into Your C++ Binary

The article discusses how a neural network can be integrated directly into a C++ binary, without the need for an inference server or API calls. The author presents a demo called dots, which places a trained neural network within a browser tab to serve as an opponent. The model consists of 2,760,322 parameters packed as fp16 and uses a 128-channel residual trunk running eleven 3x3 convolutions across a 24x24 board for each position it evaluates.

The author chose C++ for its ease of integration and portability, as it allows the model to be integrated into the same repository, build, review, and test environment as the application. The design brief for UchenML is to create small, fast models that can be easily embedded in any environment, whether it's the browser tab, inside an app, or a cloud service. This approach enables heterogeneous, distributed intelligence.

To achieve this, UchenML is built using C++20 and Bazel, and tested on various platforms including Linux, macOS, Windows, and under Emscripten for WebAssembly. The compute backend is hand-written portable SIMD code that runs on AVX-512, NEON, and WASM SIMD without requiring architecture-specific branches in the library.

The model itself is defined as a constexpr variable and is passed to a ModelParameters object, which supplies the weights. Inference is performed by calling the model with the input and parameters, and an optional scratch space can be preallocated to make the forward pass allocation-free. The model is designed to be a pure function of its input and parameters, allowing for reuse of a context across calls in a thread-safe manner.