Training Transformers on 'Inference-Only' NPUs: How VKORE Reached 0.9999 Parity
TL;DR: NPUs are sold as inference-only accelerators. VIDRAFT built VKORE, a software acceleration layer that leaves the chip untouched and (1) raises single-user inference 3.8x and server throughput 4.1x on the same NPU server, and (2) runs real transformer LoRA training on that NPU at 0.9999 numerical parity with standard PyTorch, about 170x faster than the same server's CPU. Measured on a…
VKORE, a software acceleration layer developed by VIDRAFT, enables inference-only NPUs to train transformer models while maintaining numerical parity with standard PyTorch. This innovation addresses the trend towards maximizing intelligence from existing silicon by allowing NPUs to handle both inference and training tasks, thereby reducing the need for separate GPU build-outs for fine-tuning.
On a Korean NPU server with eight cards, VKORE increased single-user inference speed from 14.1 to 53.9 tokens per second (3.8x) and server throughput from 214 to 881 tokens per second (4.1x). Crucially, this performance boost was achieved without altering the underlying hardware, utilizing only the VKORE software layer. Surprisingly, VKORE demonstrated the capability to run real transformer LoRA training at 0.9999 numerical parity with PyTorch, achieving this at approximately 170 times the speed of the server's CPU.
This breakthrough represents a significant shift in how data centers can optimize their resources, as NPUs, known for their power efficiency in inference tasks, can now also contribute to model training, thereby extending their utility and reducing overall operational costs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
