Reduce RAG costs on Amazon Bedrock with query-aware compression
Input tokens are often a meaningful part of the cost of running Retrieval Augmented Generation (RAG) at scale. This post describes a query-aware context compression pattern on Amazon Bedrock: after retrieval, a smaller model filters retrieved chunks against the query before the primary model answers, reducing input tokens and cost while preserving answer quality.
Amazon Bedrock enables developers to build Retrieval Augmented Generation (RAG) applications by providing foundation models and features. RAG typically prioritizes high recall, returning a large number of potentially relevant chunks for the primary model to work with. However, this approach can lead to high input token costs as the number of input tokens grows with the scale of workloads.
To address this, query-aware compression offers a way to reduce the number of tokens sent to the primary model while maintaining answer quality.
The compression process involves a smaller, lower-cost model that filters the retrieved chunks against the user's query before sending the context to the primary model. This results in significant cost savings while preserving answer quality. Amazon Bedrock supports custom post-retrieval processing steps like query-aware compression, which are compatible with RAG retrievers, including Amazon Bedrock Knowledge Bases.
By implementing this pattern, developers can optimize the cost-performance tradeoff when running RAG applications at scale.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
