Hi, we have an interesting issue with Typesense an...
# community-help
a
Hi, we have an interesting issue with Typesense and Kubernetes. We are using this great operator, but the issue is more Typesense related. Sometimes, our cluster is scaling down to find again a quorum, and we go from 3 to 1 node. We are parsing the node list into the container, and this list is also up-to-date. After the two nodes were down, Typesense still tried to connect to the old IPs:
Copy code
W20250610 06:48:09.540362   266 node.cpp:1559] node default_group:10.52.1.20:8107:8108 request PreVote from 10.52.19.10:8107:8108 error: [E110]Fail to connect Socket{id=764504179018 addr=10.52.19.10:8107} (0x0x7f9bfc004c00): Connection timed out
W20250610 06:48:09.540470   276 node.cpp:1559] node default_group:10.52.1.20:8107:8108 request PreVote from 10.52.2.30:8107:8108 error: [E110]Fail to connect Socket{id=755914244417 addr=10.52.2.30:8107} (0x0x7f9bfc003580): Connection timed out
I20250610 06:48:11.241796   214 raft_server.cpp:683] Term: 11, pending_queue: 0, last_index: 1822148, committed: 0, known_applied: 1821784, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 37509018
Also, after restarting the last node, he still tries to connect to the old nodes. So this is cached somewhere. Why is this happening and how can I clear this cache?
👍 1
a
Hey Adriaan, Typesense automatically polls the nodelist file every 30 seconds. If that file reflects only one node, the cluster will switch to single-node mode without needing a restart. This behavior suggests the issue might not be inside Typesense itself but rather with how the nodelist is being updated inside your pod. A few things to check: - Is the updated nodelist being written to the same file path that Typesense was started with (``--nodes`)? If the file is being replaced via Kubernetes mechanisms (e.g. ConfigMap mount), make sure the container sees the new version correctly. Sometimes volume mounts or atomic symlinks may not propagate changes the way you'd expect inside a running container. - Can you
exec
into the running container and
cat
the file? Make sure the content matches what you expect. - If you're parsing and generating this file dynamically, ensure there is no delay or race condition between the Kubernetes state change and the file update inside the container. Typesense doesn’t cache peer IPs across restarts or hold onto IPs that are no longer present in the nodelist file. If it’s still trying to connect to stale IPs, either: 1. The file hasn’t been updated properly inside the container 2. Or, it is being overwritten back to the older list by something like an init container or lifecycle hook
k
Hi folks, maintainer of the operator here. The condition has been raised as an issue again about 10d ago in GH, but I haven’t figured out any solution yet. As long as @Alan Martini confirms that TS is not caching any IPs, there might be a 3rd option: Kubernetes CoreDNS returns stale IPs (unlikely but worth taking a shot to investigate it) because the contents of the ConfigMap are not IP addresses but the FQDNs of the Pods of the Statefulset. There is no hook or init container to reset or overwrite the contents of the ConfigMap. I’ll give it a try and test a version with the IPs in the ConfigMap but due to their volatile nature in Kubernetes it would be great if we could feed braft with FQDNs.
a
Hi, thank you both for your reply 🙂. Actually, the file is properly updated. @Kyriakos Akriotis I have maybe an idea and will respond on GitHub.