When running typesense in kubernetes (perhaps using https://github.com/typesense/typesense-kubernetes if that's the official way to go) , if you use the recommended 3 node Typesense cluster (the words "node" and "cluster" here not referring to the kubernetes concepts of the same name), it seems like you would be limited to vertical scaling (a bigger machine) of the underlying (kubernetes) nodes to achieve higher throughput if your 3 (typesense) node cluster hits a performance limit. Am I mistaken here? I see that you can do 5 node typesense clusters but the 3 node setup seems to be the most common. Would it be possible to run multiple 3 node (typesense) clusters to achieve horizontal kubernetes scaling like you would by having replicas? I imagine you would need to keep the attached volumes synchronized somehow since you can't index to all locations at once. I realize also that a good answer to resolve this issue is to use Typesense cloud 🙂 .. but that wouldn't satisfy my curiosity for the moment 😄

If the primary goal is to scale read throughput, then you could add odd number of nodes into the Typesense cluster and load-balance requests between these nodes. However, the tradeoff is that each node that you add to the cluster increases write latency, because more nodes need to ack the write before the write API calls is deemed a success

Hey Jason, your quick responses are hugely appreciated, as always.

If the primary goal is to scale read throughput, then you could add odd number of nodes into the Typesense cluster and load-balance requests between these nodes. However, the tradeoff is that each node that you add to the cluster increases write latency, because more nodes need to ack the write before the write API calls is deemed a success

For our use case, write latency is not very important. We add anywhere from 5-15k documents in a single batch and can do 1 or more batches per day. Initially documents are set in an "unpublished" state by using a scoped api key and a boolean "published" field. We then upsert the batch again to publish once we've previewed/reviewed the data for accuracy. (we're not actually doing keyword search, but rather using typesense to build a search-powered experience with facets, breadcrumbs, infinite scroll etc in an easy manner with fast response times). Is increased latency linear for each added noded? I think we need to use aliases here to switch over to a fully indexed collection but I think every collection would technically double the required memory for typesense, at least temporarily. I have not yet had the chance to run full benchmarks, our production document count will start around 5 million.

I haven't tried to do this, but I'd be surprised if this works, because each cluster maintains its own node state information

Understood, that makes sense.

Another way to horizontally scale would be to shard the data among multiple clusters. So they're standard independent Typesense clusters, but on the application side, you can write certain types of records or certain user records into particular clusters

In our case everything needs to be identical, the same data is served everywhere Thinking about this as I write, this could actually make sense. Although it would require a request to the database to know which typesense cluster to query for a particular subset of the data.

Is increased latency linear for each added noded?

I haven't benchmarked the growth of latency vs node count myself. Would be curious to know what you see!

I think we need to use aliases here to switch over to a fully indexed collection but I think every collection would technically double the required memory for typesense

Yeah, for 5M documents that might be large enough RAM especially across multiple nodes for it to be expensive. So an in-place update might be better.

I haven't benchmarked the growth of latency vs node count myself. Would be curious to know what you see!

I'll keep you posted with what we see!