I Tested Amazon S3 Vectors' New Pre-Filtering Against Exact Ground Truth

The article discusses the performance comparison between AWS S3 Vectors Enhanced index mode and the legacy CLASSIC mode in handling filtered vector queries. The key findings are:

1. In the most selective filters (with 5 or fewer matching vectors), the Enhanced mode returned all exact matches for every K value (10, 20, 50) tested, while the post-filtering approach using CLASSIC mode failed to retrieve all matches, especially at higher K values.

2. For filters with 50 or more matching vectors, the Enhanced mode showed a measurable drop in recall (the proportion of true matches found) compared to the unfiltered search, indicating some accuracy trade-off for broader filters.

3. The post-filtering baseline, which applies the filter after retrieving top-K results unfiltered, performed significantly worse than Enhanced, particularly when using large budgets (10,000 or 1,000 results) for the post-filtering step.

4. The main advantage of Enhanced mode is that it finds matching vectors first by filtering, and then performs the actual vector search only on those filtered results. This eliminates the issue of the unfiltered search potentially returning fewer than the top K results, as occurs in CLASSIC mode.

5. AWS claims that Enhanced mode can return up to 5 times more matching vectors than the same query returned before Enhanced on CLASSIC indexes, but this is a ratio of result counts, not a direct Recall@K improvement.

The article concludes by emphasizing that the changes in Enhanced mode address a specific failure case in CLASSIC indexes where selective filters yield incomplete results, and the new pre-filtering approach provides more consistent recall performance, especially for highly selective queries.