I Took the Wheel Off My RAG: Managed KB vs. Manual Chunking

In April, a benchmark was published comparing five chunking strategies for Amazon Bedrock Knowledge Bases. The conclusion was straightforward: using FIXED_SIZE with S3 Vectors as the backend is optimal, unless your data justifies a more complex setup. However, three months later, AWS eliminated the need for users to choose a chunking strategy altogether. Amazon Bedrock Managed Knowledge Base handles parsing based on document type, manages the vector store, and uses AgenticRetrieveStream for complex queries.

The author conducted a study to determine the cost of quality lost when relinquishing control over the chunking process and under what conditions. The study focused on three models: Managed KB with fixed size, Managed KB with Smart Parsing, and AgenticRetrieveStream with the default planner. The author also compared the performance of these configurations against two question sets: the original 25 single-hop questions from April and 15 new multi-hop questions written against the actual document text.

The results showed that Managed KB with Simple Retrieve performed similarly to the manual configuration in accuracy. Smart Parsing was able to ingest PDFs that failed in the April configuration and did so more quickly. However, the AgenticRetrieveStream with the default planner did not generate sub-queries across all 80 evaluated questions, even when the questions were demonstrably solvable with simple retrieval.

On multi-hop questions, the AgenticRetrieveStream landed below simple retrieval, and with a small custom planner, the real decomposition was only observed on questions that did not need it. With a large custom planner, multi-hop correctness jumped from 0.50 to 1.00, but the planner and generator shared the same model. The study highlights the importance of measuring the loss of control when using Managed Knowledge Base and the potential benefits of a large custom planner.