#community-help

Troubleshooting Typesense Cluster Sync Issues in K8s Environment

TLDR Robert encountered node syncing issues in the K8s cluster with Typesense nodes. Kishore Nallan offered advice, including using DNS names and suggesting a call. Robert plans to use a single region Typesense with Redis caching. The thread resolved with the possibility of configuring Typesense with DNS names from 0.21.0.

25mo

Join the chat

Aug 17, 2021 (25 months ago)

Photo of md5-a0aba3e46685345ae57c342a8130989f

Robert

05:43 AM

Kishore Nallan we do have a stateful set within our K8s cluster with 5 typesense nodes. For some reasons when scalling down or up sometimes we get this error where the typesense nodes cannot sync between them, as the nodes don’t get config file, trying still to connect to the old one (cached somewhere with some old ips). Do you have any insights on this?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

05:45 AM

How many nodes are scaled down or up at a time? If the cluster loses quorum then this type of issue can happen.

05:47

Kishore Nallan

05:47 AM

And exact error log snippet will be useful.

Robert

06:09 AM

scaled from 5 to 7 and got this issue, also when scaled from 1 to 5

06:09

Robert

06:09 AM

will do again and send some logs

04:17

Robert

04:17 PM

another question meanwhile. We do have 8 isolated clusters in 8 regions around the world, each of it with 5 typesense nodes. So 40 typesense nodes in total. Is there possible to replicate data across regions? As right now if we have one user signing up on the app on one cluster (let’s say the Amsterdam one), it will get indexed into the 5 typesense nodes on the Amsterdam k8s clusters, but not on the others, so people from another region won’t be able to find it while searching

Kishore Nallan

04:19 PM

How much data are these collections storing?

1. Number of records and memory foot print.
2. How frequently are the records updated?
3. Streaming updates or bulk updates?

Robert

04:25 PM

streaming updates, most writes are based on registered users. it can be a few millions in the first weeks. other than that we have around 100,000 more entries, adding around 1,000 new records per day
after they are added very rare they will change (only if the user is changing the name or username)

04:26

Robert

04:26 PM

we have 8cpu 16 gb ram for each node so around 80gb of ram and 40 vcpus per cluster, so 640gb ram and 320 vcpus in total for the 8 regions

04:27

Robert

04:27 PM

cluster meaning k8s cluster

Kishore Nallan

04:42 PM

Ideally, the Typesense cluster itself should be cross-regional, which allows you rely on the built-in clustering that Typesense provides. But it's probably difficult to create a cross regional k8s setup this way and ensure routing of request to the nearest node etc., In general cross regional clustering is pretty challenging to pull off but it's available in Typesense Cloud if you wish to consider that.

Another option will be to use a kafka cluster where all writes are written to, and then you have 1 consumer in each region to index independently.

Robert

04:54 PM

we do have right now around 1.6M records in one typesense cluster. with that 5 nodes of 8vcpu and 16gb ram each we can get around 5,000 rps. so with 8 clusters we would get around 40,000 rps globally. How big of a cluster do you think we would need on your Cloud solution to reach same results?

Aug 18, 2021 (25 months ago)

Kishore Nallan

05:22 AM

It will be easier to get into the details over a quick call. Would you be free sometime today or tomorrow? My calendar: https://calendly.com/kishorenc/30min?month=2021-08

11:47

Kishore Nallan

11:47 AM

Robert (forgot to @ you earlier and not sure if Slack would alert you otherwise)

Robert

11:48 AM

hi Kishore, for now we chose to keep typesense only on one Region and all the other regions will connect to it, and we’ll just cache the results in Redis for 1h to have it working fast

Kishore Nallan

11:49 AM

👍 Happy to chat anytime if you want to consider alternatives going forward.

Robert

11:49 AM

question