#community-help

Configuring a Typesense Cluster

TLDR Chetan had trouble configuring a Typesense cluster and facing issues with "peering-address". Kishore Nallan advised defaults for network settings, understanding Docker's role in networking, and provided tips for troubleshooting. Issue resolved when Chetan used a specific Docker argument.

Powered by Struct AI
34
3mo
Solved
Join the chat
Aug 23, 2023 (3 months ago)
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
06:05 AM
some questions regarding the HA guide:
• what should be put in "peering-address"? is it required?
◦ 0.0.0.0 /internal IPs/external IP addresses trigger "failed to listen" errors
◦ leaving it blank starts, but i think typesense sees its own IP in the peers as a different peer with whom it cannot communicate, so it doesn't elect itself leader
• do the instructions for HA assume all the processes are running in the same node? i ask because they all seem to use 192.168-based IPs which are all local to a single machine?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:12 AM
Due to some recent changes necessiated for cross cluster communication, peering address can only be 0.0.0.0 -- (default) so I recommend not setting it explicitly.
06:12
Kishore Nallan
06:12 AM
> do the instructions for HA assume all the processes are running in the same node? i ask because they all seem to use 192.168-based IPs which are all local to a single machine?
No, you could have internal IPs that begin with 192.168 for separate machines as well.
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
06:23 AM
i'm a bit stumped here then as to why i can't get this node to identify itself as the leader - it should according to the HA guide:
Force one of the nodes to become a single-node cluster by editing its nodes file to contain just its own IP address
Image 1 for i'm a bit stumped here then as to why i can't get this node to identify itself as the leader - it should according to the HA guide:
```Force one of the nodes to become a single-node cluster by editing its nodes file to contain just its own IP address```
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:24 AM
If you have some older configuration which was using peering address, you should probably stop, clear the data and dir and start fresh as some core configurations could persist and prevent a reset.
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
06:27 AM
clear all data? or just one of the folders?
06:28
Chetan
06:28 AM
Image 1 for
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:29 AM
State is the only directory that mattera. Other two are populated from state.
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
06:31 AM
didn't seem to resolve everything - at this point i'm just starting an empty server
Image 1 for didn't seem to resolve everything - at this point i'm just starting an empty server
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:40 AM
Can you post your full config file?
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
03:23 PM
Image 1 for
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:24 PM
Docker networking is the other variable here.
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
03:27 PM
yup, and VM networking. but i've verified those are working because i can ping port 8107 from one VM to the other and get back results that seem correct
03:28
Chetan
03:28 PM
here's a curl from one node to port :8107 of the others (and itself)
Image 1 for here's a curl from one node to port :8107 of the others (and itself)
08:23
Chetan
08:23 PM
just verified the curl commands work inside the container too
Image 1 for just verified the curl commands work inside the container tooImage 2 for just verified the curl commands work inside the container too
10:06
Chetan
10:06 PM
so they do seem to be talking to each other (i think), it's just the one node talks to another saying "X command is for node 10.0.0.5:8107" and the server responds "yo idk who 10.0.0.5:8107 is, i'm 172.17.0.2:8107:443" (which is a local IP - they're all that specific port funny enough).
Image 1 for so they do seem to be talking to each other (i think), it's just the one node talks to another saying "X command is for node 10.0.0.5:8107" and the server responds "yo idk who 10.0.0.5:8107 is, i'm 172.17.0.2:8107:443" (which is a local IP - they're all that specific port funny enough).
Aug 24, 2023 (3 months ago)
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
04:27 AM
hi Kishore Nallan - any chance the above screenshot gives any insights on what may be up here?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:28 AM
Can you post the contents of the nodes file also
04:31
Kishore Nallan
04:31 AM
Nodes file should not contain 0.0.0.0 -- it should have internal IPs of the nodes in the cluster.
04:33
Kishore Nallan
04:33 AM
The other suggestion I have is to specify a peering address but given the internal IP. In the screenshots, I see both 10. and 172. getting used so I wonder if there are two networks and Typesense is not picking the correct IP for cluster peering.
04:33
Kishore Nallan
04:33 AM
So give a --peering-address=10.x.x.x explicitly.
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
04:47 AM
here's the current file - just trying to get the node to identify itself as the leader.
Image 1 for here's the current file - just trying to get the node to identify itself as the leader.
04:48
Chetan
04:48 AM
and yes it doesn't seem to recognize 10.0.0.4 as itself
Image 1 for and yes it doesn't seem to recognize 10.0.0.4 as itself
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:48 AM
Try using the 172.x address
04:49
Kishore Nallan
04:49 AM
Also delete data directory and start fresh
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
04:51 AM
the 172.x address is the same though across my VMs - all of them seem to self-identify is 172.17.0.2:8107
04:51
Chetan
04:51 AM
telling them all --peering-address 172.17.0.2:8107 seems wrong?
04:52
Chetan
04:52 AM
here's what happens when i try telling it its IP is 10.0.0.5 though
Image 1 for here's what happens when i try telling it its IP is 10.0.0.5 though
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:52 AM
Hmm, sorry I am not familiar with Docker networking enough to help further here.
04:53
Kishore Nallan
04:53 AM
Does ifconfig list that 10.x IP?
Chetan
Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20
Chetan
04:58 AM
ah no worries, i didn't realize that docker was causing the failure to listen to 10.0.0.5
04:58
Chetan
04:58 AM
i assumed it was typesense itself.
04:59
Chetan
04:59 AM
the fix for docker was simple - i just added a --network host argument (not the most security-ideal solution, but works for this case)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:01 AM
👍

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting Multi-Node Setup in Docker

Harpreet experienced issues running a multi-node setup on Docker and received troubleshooting advice and alternative solution from Kishore Nallan.

2

13
31mo

Setting Up HA Cluster on Fly with Typesense

Jordan needed help setting up an HA cluster on Fly for Typesense. Jason assisted in troubleshooting the configuration and 6tunnel usage. Eventually, a solution was found in the Fly community forum.

1

49
6mo
Solved

Troubleshooting Typesense Cluster Multi-node Leadership Error

Bill experienced a problem with a new typesense cluster, receiving an error about no leader and health status issues. Jason and Kishore Nallan provided troubleshooting steps and determined it was likely due to a communication issue between nodes. Kishore Nallan identified a potential solution involving resetting the data directory. Following this, Bill reported the error resolved.

45
24mo
Solved

Testing High Availability with Raft Returns Crashes

pboros reports an issue with usual crashes when testing high availability with Raft. Kishore Nallan suggests checking the quorum recovery period and efficiently logging the crash on all nodes. The issue persists, with pboros suspecting it's due to hostname being no longer resolvable once a container is killed.

33
17mo

Debugging and Recovery of a Stuck Typesense Cluster

Charlie had a wedged staging cluster. Jason provided debugging and recovery steps, and Adrian helped with more insights. It turns out the issue was insufficient disk space. Once Adrian increased the disk size, the cluster healed itself.

6

55
4mo
Solved