Installing Rust for vLLM on Graviton: a G5g walk-through 🦀

This guide walks readers through the process of installing the Rust toolchain for vLLM on an AWS EC2 Graviton2 (aarch64) instance with an NVIDIA T4G GPU. The article explains that vLLM now requires Rust to build, as it is a 14-crate Rust workspace that includes the HTTP frontend, tool parser, tokenizer, and benchmark client. The guide highlights the differences between building vLLM from source and installing it via pip, emphasizing that pip automatically installs setuptools_rust, torch, and wheel, while building from source requires manual installation of these dependencies.

The tutorial begins by installing Rust using rustup, which successfully installs a stable version of Rust for aarch64 architecture. Next, it installs setuptools_rust using python3 -m pip install setuptools_rust, ensuring it is installed in the same interpreter used for building. The guide then installs the protobuf compiler with apt-get, which is required for the gRPC stubs in vllm-server crate.

It also installs the CUDA toolkit for vLLM's kernels, noting that the NVIDIA sbsa repository is used for the aarch64 architecture. Finally, the guide builds vLLM with setuptools-rust, specifying the --release flag to build optimized release artifacts instead of debug ones.