#community-help

Troubleshooting Typesense Cluster Sync Issues in K8s Environment

TLDR Robert encountered node syncing issues in the K8s cluster with Typesense nodes. Kishore Nallan offered advice, including using DNS names and suggesting a call. Robert plans to use a single region Typesense with Redis caching. The thread resolved with the possibility of configuring Typesense with DNS names from 0.21.0.

Powered by Struct AI
Aug 17, 2021 (25 months ago)
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
05:43 AM
Kishore Nallan we do have a stateful set within our K8s cluster with 5 typesense nodes. For some reasons when scalling down or up sometimes we get this error where the typesense nodes cannot sync between them, as the nodes don’t get config file, trying still to connect to the old one (cached somewhere with some old ips). Do you have any insights on this?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:45 AM
How many nodes are scaled down or up at a time? If the cluster loses quorum then this type of issue can happen.
05:47
Kishore Nallan
05:47 AM
And exact error log snippet will be useful.
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
06:09 AM
scaled from 5 to 7 and got this issue, also when scaled from 1 to 5
06:09
Robert
06:09 AM
will do again and send some logs
04:17
Robert
04:17 PM
another question meanwhile. We do have 8 isolated clusters in 8 regions around the world, each of it with 5 typesense nodes. So 40 typesense nodes in total. Is there possible to replicate data across regions? As right now if we have one user signing up on the app on one cluster (let’s say the Amsterdam one), it will get indexed into the 5 typesense nodes on the Amsterdam k8s clusters, but not on the others, so people from another region won’t be able to find it while searching
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:19 PM
How much data are these collections storing?

1. Number of records and memory foot print.
2. How frequently are the records updated?
3. Streaming updates or bulk updates?
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
04:25 PM
streaming updates, most writes are based on registered users. it can be a few millions in the first weeks. other than that we have around 100,000 more entries, adding around 1,000 new records per day
after they are added very rare they will change (only if the user is changing the name or username)
04:26
Robert
04:26 PM
we have 8cpu 16 gb ram for each node so around 80gb of ram and 40 vcpus per cluster, so 640gb ram and 320 vcpus in total for the 8 regions
04:27
Robert
04:27 PM
cluster meaning k8s cluster
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:42 PM
Ideally, the Typesense cluster itself should be cross-regional, which allows you rely on the built-in clustering that Typesense provides. But it's probably difficult to create a cross regional k8s setup this way and ensure routing of request to the nearest node etc., In general cross regional clustering is pretty challenging to pull off but it's available in Typesense Cloud if you wish to consider that.

Another option will be to use a kafka cluster where all writes are written to, and then you have 1 consumer in each region to index independently.
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
04:54 PM
we do have right now around 1.6M records in one typesense cluster. with that 5 nodes of 8vcpu and 16gb ram each we can get around 5,000 rps. so with 8 clusters we would get around 40,000 rps globally. How big of a cluster do you think we would need on your Cloud solution to reach same results?
Aug 18, 2021 (25 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:22 AM
It will be easier to get into the details over a quick call. Would you be free sometime today or tomorrow? My calendar: https://calendly.com/kishorenc/30min?month=2021-08
11:47
Kishore Nallan
11:47 AM
Robert (forgot to @ you earlier and not sure if Slack would alert you otherwise)
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
11:48 AM
hi Kishore, for now we chose to keep typesense only on one Region and all the other regions will connect to it, and we’ll just cache the results in Redis for 1h to have it working fast
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:49 AM
👍 Happy to chat anytime if you want to consider alternatives going forward.
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
11:49 AM
question
11:49
Robert
11:49 AM
Configuring a Typesense cluster
11:50
Robert
11:50 AM
for the config file
11:50
Robert
11:50 AM
can we use dns names instead of ips?
11:50
Robert
11:50 AM
192.168.12.1:8107:8108,192.168.12.2:8107:8108,192.168.12.3:8107:8108
11:50
Robert
11:50 AM
here
01:44
Robert
01:44 PM
Kishore Nallan ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:45 PM
Sorry I didn't get an alert for your messages. Yes you can use DNS names from 0.21.0
Robert
Photo of md5-a0aba3e46685345ae57c342a8130989f
Robert
03:05 PM
we do have this issue where the 5 typesense nodes that we are using are not starting at the exact same time so it taskes some time for all of them to get the right config file and work with all the 5 nodes running
03:06
Robert
03:06 PM
did you encounter this before or do you have any insights on that? Kishore Nallan
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:07 PM
How long does it take? I don't have operational experience running Typesense on Kubernetes.