@Kishore Nallan we do have a stateful set within our K8s cluster with 5 typesense nodes. For some reasons when scalling down or up sometimes we get this error where the typesense nodes cannot sync between them, as the nodes don’t get config file, trying still to connect to the old one (cached somewhere with some old ips). Do you have any insights on this?

How many nodes are scaled down or up at a time? If the cluster loses quorum then this type of issue can happen.

scaled from 5 to 7 and got this issue, also when scaled from 1 to 5

How much data are these collections storing? 1. Number of records and memory foot print. 2. How frequently are the records updated? 3. Streaming updates or bulk updates?

streaming updates, most writes are based on registered users. it can be a few millions in the first weeks. other than that we have around 100,000 more entries, adding around 1,000 new records per day after they are added very rare they will change (only if the user is changing the name or username)

Ideally, the Typesense cluster itself should be cross-regional, which allows you rely on the built-in clustering that Typesense provides. But it's probably difficult to create a cross regional k8s setup this way and ensure routing of request to the nearest node etc., In general cross regional clustering is pretty challenging to pull off but it's available in Typesense Cloud if you wish to consider that. Another option will be to use a kafka cluster where all writes are written to, and then you have 1 consumer in each region to index independently.

we do have right now around 1.6M records in one typesense cluster. with that 5 nodes of 8vcpu and 16gb ram each we can get around 5,000 rps. so with 8 clusters we would get around 40,000 rps globally. How big of a cluster do you think we would need on your Cloud solution to reach same results?

It will be easier to get into the details over a quick call. Would you be free sometime today or tomorrow? My calendar: https://calendly.com/kishorenc/30min?month=2021-08

hi Kishore, for now we chose to keep typesense only on one Region and all the other regions will connect to it, and we’ll just cache the results in Redis for 1h to have it working fast

👍 Happy to chat anytime if you want to consider alternatives going forward.

question

Sorry I didn't get an alert for your messages. Yes you can use DNS names from 0.21.0

we do have this issue where the 5 typesense nodes that we are using are not starting at the exact same time so it taskes some time for all of them to get the right config file and work with all the 5 nodes running

How long does it take? I don't have operational experience running Typesense on Kubernetes.