Optimizing cost and latency with Amazon Bedrock prompt caching
Prompt caching in Amazon Bedrock can cut input token costs by up to 90% when you repeatedly send the same context to foundation models. This post walks through six practical prompt caching scenarios using the Converse API: message content, system prompt, tool definition, mixed TTL, tenant isolation, and LangChain integration.
Amazon Bedrock's prompt caching feature can significantly reduce text input costs by up to 90% when you send the same context to foundation models repeatedly. Without caching, billing full price for content already processed results in 500,000 input tokens for a 10,000-token contract and 50 user questions. Options to mitigate this issue include shortening prompts, reducing context windows, or implementing application-level caching.
Each option involves trade-offs, but prompt caching at the infrastructure level can help reduce time-to-first-token (TTFT) and lower costs for cached input tokens by up to 90% on cache hits without changing the model or prompt quality. Six practical scenarios using the Converse API in Amazon Bedrock illustrate prompt caching: message content caching, system prompt caching, tool definition caching, mixed TTL caching, tenant isolation, and LangChain integration.
Prompt caching stores a snapshot of partially processed input so subsequent requests with the same prefix skip redundant computation. Cache entries are scoped to individual AWS accounts and regions, with token thresholds for activation and time-to-live (TTL) based on the request. Model-agnostic syntax is identical across supported model families.
Pricing introduces two token categories: cacheWriteInputTokens (25% higher) and cacheReadInputTokens (90% lower). This can result in approximately 75% savings on input token costs for workloads with repeated context.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
