Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU
A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g * — a Graviton2 (aarch64) host with an NVIDIA T4G (Turing, SM 7.5) GPU. Three obstacles: an arch list nobody publishes for this combination, a version floor that only the newest vLLM clears, and 64 KiB of shared memory that stops the model dead. Plus the seven things I documented wrong before I had a box.* Model google/gemma-4-E2B-it…
The piece reports on running Google's Gemma 4 model on Amazon Web Services (AWS) infrastructure, specifically on an EC2 G5g instance with a Graviton2 processor and an NVIDIA T4G GPU. The article highlights three main challenges: lack of documentation for this specific combination, a version floor that only supports the latest vLLM release, and 64 KiB of shared memory which restricts the model's performance.
The author also points out several issues encountered during the setup, such as missing compiler and support for aarch64 architecture with SM 7.5 instruction set, as well as the need for a Rust toolchain for vLLM to function correctly.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
