Open Source pg_vectorsearch Brings More Scalable Vector Search to PostgreSQL

Tiger Data has introduced pg_vectorsearch, an open-source PostgreSQL extension designed for scalable vector similarity search, currently in its alpha phase. The extension, named after Predator, the company's headline product, aims to provide faster and more memory-efficient vector search directly within PostgreSQL, minimizing infrastructure requirements for managing extensive vector datasets.

Utilizing the PRISM index, pg_vectorsearch employs data partitioning and compression techniques to enhance performance, as opposed to the graph-based HNSW index utilized by pgvector.

Developers can integrate these capabilities into their applications by installing the extension and generating a PRISM index on vector columns. According to Tiger Data's benchmarks, the PRISM index can build an index for 100 million vectors in approximately 1 hour and 40 minutes on a machine equipped with 128 GB of memory, whereas the HNSW index took 42 hours on a machine with 512 GB of memory.

Erik Nordström, a senior staff engineer at Tiger Data, Matvey Arye, a principal engineer at the company, and Noah Hein, a staff product marketing manager at Tiger Data, describe the PRISM approach as a partition-based method similar to those found in newer dedicated vector search systems. Instead of establishing connections between every vector and its neighbors in a graph, PRISM clusters similar vectors into partitions and stores them as compressed lists within database pages.

This technique minimizes index size and eliminates the need for costly random memory or disk access, thereby enabling quicker vector search with reduced RAM requirements in PostgreSQL.

Michael Freedman, Tiger Data's co-founder and chief technology officer, published an article detailing the architecture of the PRISM vector index. The index blends hierarchical clustering to narrow the search space, compressed vectors for efficient sequential scanning, and RaBitQ quantization to minimize resource-intensive distance computations.

To maintain search efficiency, pg_vectorsearch duplicates vectors near partition boundaries, enhancing recall while relying on PostgreSQL's inherent storage, logging, and concurrency mechanisms. However, the system does not yet support automatic rebalancing of the index, implying that frequent data modifications might necessitate rebuilding the index to preserve search efficacy.

While the extension offers promising advantages, including reduced memory requirements and faster search times, it remains in its alpha stage and is not yet recommended for production use. Practitioners acknowledge the potential benefits but caution about the need for independent benchmark validation and the possibility of pg_vectorsearch replacing pgvectorscale.

Furthermore, the extension lacks support for filtered searches, which are expected to be added before its general availability. The project's code is accessible on GitHub, and Tiger Data plans to transition its existing pgvectorscale extension into maintenance mode.