API Gateway HTTP API + Lambda Function URLs: Building a Prompt‑Caching LLM Proxy, Explained Simply

Large language model (LLM) features such as code completion typically reside inside heavyweight Lambda functions, which introduce latency and increase costs. To address this, a technique called prompt caching is employed. By intercepting repeated prompts and storing the model's response for quick retrieval, the overall response time for the user is significantly reduced.

A prompt cache operates like a sticky note: the first time a user asks a question, the answer is computed and stored. On subsequent identical queries, the answer is retrieved instantly from the cache, bypassing the need to recompute the model's response. This approach saves both time and money, as it eliminates redundant computations.

Storing the cache within the Lambda function rather than an external data store offers several benefits. Firstly, it provides faster access, as the map object used for caching resides within the same process and lookup times are minimal. Secondly, it reduces costs by avoiding additional reads from a separate database or Redis instance. Lastly, it enhances simplicity by minimizing the need for extra IAM policies and VPC configurations.

The main drawback of this method is that the cache is lost when the Lambda container is terminated. However, for prompts that are frequently requested, this is a negligible trade-off, as the next cold start will promptly rebuild the cache.

The key takeaway is that by implementing in-process caching at the function level, tens of milliseconds can be shaved off each repeated request, all without the need for a separate caching service.

To set up this prompt caching mechanism, API Gateway’s HTTP API (v2) is utilized, which offers a lighter and cheaper alternative compared to the traditional REST API (v1). The HTTP API supports Function URLs, enabling a direct HTTPS endpoint that communicates with the Lambda function without the need for additional integration layers.

Function URLs provide several advantages over the traditional Lambda integration approach. They result in less latency, as the request goes directly to the Lambda without passing through an extra JSON mapping layer. Additionally, Function URLs simplify permissions management, requiring only the API's principal to be authorized to call the URL. Furthermore, they are more cost-effective, being priced based on the number of requests rather than incurring per-request surcharges.

The following is a concise script written in Node.js 22 using the AWS SDK that demonstrates the creation of an HTTP API with a route specifically designed to cache prompts. This script leverages the `ApiGatewayV2Client` and employs various commands to establish the necessary components:

1. **Create an HTTP API**: The script initiates an HTTP API with the name 'PromptCacheAPI', using the HTTP protocol type. It also enables binary support, allowing payloads larger than 6 KB to be successfully transmitted to the Lambda function, thereby avoiding the hidden 6 KB body drop bug.

2. **Create an Integration**: An integration point is configured to direct requests from the HTTP API to the specified Lambda Function URL. This integration is set to HTTP_PROXY mode, ensuring that the request is forwarded to the Lambda without any transformation.

3. **Create a Route**: A route is established within the HTTP API, mapping to the endpoint `/prompt`. This route is configured to target the previously created integration, ensuring that any incoming POST requests to `/prompt` are routed to the Lambda function encapsulated by the Function URL.

4. **Optional CORS Configuration**: Although not mandatory for all use cases, enabling Cross-Origin Resource Sharing (CORS) is recommended. This step configures the API to allow requests from any origin, supporting methods limited to POST, and specifying the necessary headers.

By executing this script, a fully functional HTTP API is generated, complete with a route capable of invoking a Lambda function via a Function URL. The endpoint for this API can be accessed at `https://<api-id>.execute-api.<region>.amazonaws.com/prompt`, where `<api-id>` and `<region>` are placeholders for the specific API ID and AWS region used during the creation process.