Why we stopped using one Elasticsearch index per tenant and built our own

We abandoned using one Elasticsearch index per tenant and instead developed our own solution called SeaSearch. This decision was driven by the limitations of replicating data across indexes, which caused operational complexity and scalability issues. SeaSearch tackles these problems by storing index data in S3 and managing ownership instead of the data itself.

The primary challenge is the cost of each index, which requires at least one shard, much like a mini Lucene database. As the number of indexes increases, so does the strain on the master node, leading to sluggish performance. Rather than building a search engine from scratch, the team chose ZincSearch as a foundation, which provided most of the necessary runtime footprint and migration capabilities.

To address the issue of unbounded index counts, SeaSearch stores indexes in S3 and routes ownership through etcd and a cluster manager. Compute nodes handle reads and writes, all communicating with S3-compatible storage. The proxy (gateway) routes client requests to the appropriate node, which then retrieves the required segments from S3 as needed. This approach eliminates the need for redundant data copies and simplifies failover processes.

One potential drawback is the speed of S3 compared to local NVMe storage. To mitigate this, SeaSearch employs a cache system on compute nodes, utilizing immutable segments for read-only data. This allows compute nodes to serve indexes larger than their local disks by continuously cycling through segments. Parallel warm-up and distributed query execution further enhance performance, distributing cache pressure and accelerating query processing across multiple nodes.