Gemma 4 on an Amazon SageMaker Endpoint: AWS CLI, NVIDIA L4, and an MCP Server

This article outlines a step-by-step guide for deploying Gemma 4 E2B to an Amazon SageMaker endpoint hosted on a single NVIDIA L4 GPU using the vLLM container provided by AWS. The deployment utilizes the vLLM container published by AWS for SageMaker, and every AWS call is made using plain AWS CLI commands, allowing for manual execution or automation through an MCP server.

A SageMaker real-time endpoint functions as a managed HTTPS inference server, automatically handling health checks, request routing, and log writing to CloudWatch without requiring manual patching, security group configuration, or load balancer setup.

To begin the MCP development for model management, a step-by-step approach is recommended. This involves setting up the basic development environment with necessary system variables and a functioning Claude Code configuration. Subsequently, the Python MCP server is initiated in stdio mode and validated using Claude Code within the local environment. The overall deployment process consists of eight steps, each demonstrated through AWS CLI commands and corresponding MCP tools.

Upon completion, the setup should include an AWS account with AWS CLI v2 installed and signed in, a SageMaker endpoint quota of at least 1 for compatible instance types (such as ml.g6.xlarge, ml.g6.2xlarge, or ml.g6.4xlarge), Python 3.11 or later, with MCP 2.x and Claude Code or Gemini CLI installed and functional, and jq for parsing JSON responses.

The environment should be configured by cloning the repository, installing required dependencies, and setting up a configuration file (.env) that stores various settings used by the tools.

The project is based on the Apache-2.0 licensed Gemma 4 E2B model, which is open for unrestricted use according to the model's license. A Python development environment is required, including the installation of mcp 2.x and Claude Code or Gemini CLI. A JSON-RPC transport mechanism is employed for communication between the client (Claude Code) and the server, which is implemented in a single file, server.py, utilizing the mcp Python SDK 2.x.

Each tool within the server communicates with the AWS CLI as a subprocess, automatically refreshing AWS login credentials and ignoring any inherited session tokens to maintain credential freshness.

A comprehensive set of tools is provided to manage the Gemma 4 deployment, including commands for checking quotas, deleting endpoints, deploying endpoints, finding the optimal vLLM image, retrieving deployment configurations, logging endpoint activities, checking endpoint statuses, listing endpoints, querying model health, and verifying model configurations. These tools collectively enable a structured approach to managing the Gemma 4 deployment on SageMaker, ensuring efficient and reliable model hosting.