Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

Developers desire AI coding assistants that are easy to set up. By using an LLM gateway with LiteLLM and Amazon Bedrock, they can access foundation models readily while maintaining control over usage and observability. Platform teams monitor latency and errors, finance teams assign inference costs to teams and projects, and security teams enforce consistent access controls.

An LLM gateway offers a centralized point of access to the models powering coding assistants. This post demonstrates building a self-service LLM gateway using Amazon Bedrock Application Inference Profiles with LiteLLM on AWS Fargate, abstracting Amazon Bedrock behind an OpenAI-compatible API. Developers access foundation models through a single API with a dedicated key, enabling per-user cost controls and model performance observability.

By the end, they will have the knowledge to build their own version using illustrative snippets and AWS Cloud Development Kit (AWS CDK) constructs. The post employs Claude Sonnet and Claude Haiku models as examples, with a list of available models and their AWS regions provided. The LLM gateway design follows a simple request flow from the developer, through the LiteLLM layer, to Amazon Bedrock and the underlying foundation model.

The diagram illustrates the production architecture. For brevity, Amazon Bedrock Guardrails such as content filtering and grounding are not covered but would typically be added for a production deployment. A developer's OpenAI-compatible client reaches the LLM gateway via Amazon Route 53, Amazon CloudFront, and an Application Load Balancer (ALB).

Authentication to LiteLLM on Fargate is performed using a per-user API key. The request accesses Amazon RDS (PostgreSQL) for virtual-key and spend state, interacts with Amazon ElastiCache for Valkey as a cache, and uses AWS Secrets Manager for secrets. LiteLLM ultimately invokes Amazon Bedrock Application Inference Profiles using a scoped AWS Identity and Access Management (IAM) task role.

The architecture consists of several core components: LiteLLM on AWS Fargate, providing an OpenAI-compatible API; a self-service portal for admin and developer configuration; persistent and cache storage using Amazon ElastiCache for Valkey and Amazon RDS (PostgreSQL); Amazon Bedrock Application Inference Profiles for each model; and Langfuse for LLM observability.

Using an OpenAI-compatible protocol, developers configure their tools with a single base URL and API key, allowing them to operate independently of the Amazon Bedrock backend. Configurations can be switched, fallbacks added, or model providers migrated without developers needing to modify their setups. Prerequisites include an AWS account, access to Amazon Bedrock models, appropriate IAM permissions, and LiteLLM.

The open-source LiteLLM AI Gateway component is suitable for a small pilot, with limitations and a license required for larger usage. LiteLLM Enterprise is a scalable, production-ready licensed version that includes the open-source gateway plus additional enterprise features. End-to-end configuration proceeds from the developer's IDE, through LiteLLM configuration (using AWS Fargate), mapping and enforcing model names to Amazon Bedrock inference profile ARNs with per-key spend limits, and finally to the foundation model.

LiteLLM, running on AWS Fargate, includes the portal, which can be accessed locally during testing. LiteLLM maps and enforces model names to Amazon Bedrock inference profile ARNs, enforcing per-key spend limits.