Nvidia's RTX Spark has an NPU it barely mentions, and I spent a week figuring out what it actually is

Nvidia's RTX Spark is an intriguing device that has caught the attention of tech enthusiasts and professionals alike. While the Nvidia RTX 30 series GPUs have been well-documented, the RTX Spark stands out due to its inclusion of a Neural Processing Unit (NPU). Initially, Nvidia only hinted at the NPU's capabilities, which were primarily geared towards Copilot+ features.

After thoroughly investigating the RTX Spark, I discovered that the NPU is a 55 TOPS accelerator, specifically designed for Copilot+ features. This NPU has its own registers, clock, and power rail, and it operates independently from the GPU. Although Windows lists it in Task Manager and Device Manager, third-party software cannot access it through normal interfaces.

The NPU's architecture is a descendant of Nvidia's Deep Learning Accelerator (DLA), featuring RISC-V processors. After conducting extensive research, I managed to run neural networks on the NPU, confirming that the GPU was not secretly performing the computations. I also successfully ran the ResNet-50 model in INT8, a mode that Nvidia claims is not supported on this platform, although the performance gains were minimal.

To determine the nature of the NPU, I delved into Nvidia's beta development tools and the RTX Spark's firmware. The firmware reveals a device called DLA0, which includes two memory-mapped register windows and ACPI logic for hiding the NPU from Windows. The device is integrated into the SoC alongside the display, camera, and audio blocks, with the GPU on a separate die connected over Nvidia's C2C link.

Upon further investigation, I discovered that the NPU is treated differently by Windows compared to a regular GPU. It uses a compute-only version of the display driver model (MCDM) and is assigned a single engine called NPU_0. The NPU also has its own independent clock domain, power consumption, and is cache-coherent with the CPU.

While Nvidia's cuDLA API, designed for Jetson accelerators, can communicate with the NPU, running actual neural networks requires Nvidia's TensorRT. However, TensorRT has several limitations, such as requiring every layer to run on the DLA without GPU fallback, a fixed batch size of one, static shapes, and computation only using FP16. Despite these restrictions, INT8 mode still functions, albeit with less performance improvement than expected.

In conclusion, the NPU in the RTX Spark is a unique and powerful component that sets this device apart from its competitors. Although Nvidia has been relatively scarce in providing details about the NPU, my investigation revealed its capabilities and potential applications. As the RTX Spark continues to gain traction, it will be interesting to see how Nvidia develops and promotes its NPU technology in the future.