Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU
A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it…
Running Gemma 4 on AWS EC2 G5g: Graviton2 with NVIDIA GPU
Three main obstacles were encountered when attempting to run Google's Gemma 4 E2B model on an AWS EC2 G5g instance, which features a Graviton2 (aarch64) host and an NVIDIA T4G (Turing, SM 7.5) GPU. The primary issues included a lack of published architecture lists for this combination, a version floor only cleared by the newest vLLM, and 64 KiB of shared memory that prevents the model from functioning properly. Additionally, seven pitfalls were documented prior to achieving successful operation.
While the G5g instance is the only AWS-hosted environment that combines an NVIDIA GPU with a Graviton host, it launched in 2020 and has not received any successors. This unique configuration, supported by NVIDIA's Grace CPU, stands out as it moved the Arm-plus-CUDA world away from NVIDIA's own Arm CPUs. The scarcity of published builds for this combination led to the exploration of alternative solutions.
The packaging issue proved relatively straightforward to resolve. However, deeper challenges arose, such as a missing compiler, a version floor not anticipated, and 32 KiB of shared memory that proved problematic. Crucially, no published build encompassed both aarch64 and SM 7.5 configurations together, necessitating further investigation.
The examination of available Docker images revealed that the vllm/vllm-openai:v0.27.1 tag contains both platforms, with architecture lists that illustrate the discrepancy between architectures supported. Unfortunately, the aarch64 list excludes Turing, which is the only architecture required by the G5g instance. This realization highlighted a critical limitation in the existing build options.
Furthermore, the ecosystem exhibited similar split behavior across various packages. Despite the availability of PyTorch 2.12 on ARM64 architecture, it had dropped support for the G5g's Turing architecture in subsequent releases. The AWS-provided CUDA wheels lacked support for the required compute capability, while the from-source PyTorch build was deemed too outdated for Gemma 4.
Upon encountering these issues, the latest vLLM release proved to be the most compatible with the G5g instance. This discovery became pivotal in overcoming the remaining obstacles, demonstrating the importance of leveraging the most recent software versions when facing compatibility challenges. The experience served as a valuable lesson in prioritizing the latest releases and clearly communicating constraints when encountering setbacks during the model deployment process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
