Amazon introduces toolkit for running PyTorch workloads on AWS Trainium chips

Amazon has launched TorchNeuron, an open-source toolkit that allows developers to run PyTorch workloads on AWS Trainium chips. PyTorch is the most widely used AI development framework. Previously, running AI projects on Trainium-based AWS instances required significant manual intervention, raising development costs and posing an entry barrier for users.

TorchNeuron simplifies this process by enabling PyTorch projects to run on Trainium chips with minimal code changes. Developers only need to configure a setting to specify the chip for the workload. TorchNeuron is compatible with Trainium2 and Trainium3, the two latest accelerators in the series. Trainium3, the more powerful processor, boasts eight computing cores, each with four modules optimized for different subsets of AI model calculations.

The chip features a 154-gigabyte onboard memory pool. AWS runs Trainium3 in custom servers, each capable of housing up to 144 accelerators. A single system can execute up to 362 quadrillion calculations per second in the MXFP8 format, a widely used data format by AI models. TorchNeuron includes NKI operators, software building blocks that help developers optimize AI models by making low-level adjustments, reducing hardware usage, and enhancing speed.

These operators, along with support for PyTorch and other open-source tools like TorchTitan, enable developers to distribute training across accelerators and optimize the training workflow. TorchNeuron has already been adopted by several AWS customers, including Cartesia AI Inc. and the Lieber Institute for Brain Development. Amazon intends to release TorchNeuron as open-source on the first day of a two-day PyTorch Conference in San Jose.