#community-help

Configuring a Typesense Cluster

TLDR Chetan had trouble configuring a Typesense cluster and facing issues with "peering-address". Kishore Nallan advised defaults for network settings, understanding Docker's role in networking, and provided tips for troubleshooting. Issue resolved when Chetan used a specific Docker argument.

1mo

Join the chat

Aug 23, 2023 (1 month ago)

Photo of md5-f5bf390d5711f8a2f96bd93d1bc97d20

Chetan

06:05 AM

some questions regarding the HA guide:
• what should be put in "peering-address"? is it required?
◦ 0.0.0.0 /internal IPs/external IP addresses trigger "failed to listen" errors
◦ leaving it blank starts, but i think typesense sees its own IP in the peers as a different peer with whom it cannot communicate, so it doesn't elect itself leader
• do the instructions for HA assume all the processes are running in the same node? i ask because they all seem to use 192.168-based IPs which are all local to a single machine?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

06:12 AM

Due to some recent changes necessiated for cross cluster communication, peering address can only be 0.0.0.0 -- (default) so I recommend not setting it explicitly.

06:12

Kishore Nallan

06:12 AM

> do the instructions for HA assume all the processes are running in the same node? i ask because they all seem to use 192.168-based IPs which are all local to a single machine?
No, you could have internal IPs that begin with 192.168 for separate machines as well.

Chetan

06:23 AM

i'm a bit stumped here then as to why i can't get this node to identify itself as the leader - it should according to the HA guide:

Force one of the nodes to become a single-node cluster by editing its nodes file to contain just its own IP address

Kishore Nallan

06:24 AM

If you have some older configuration which was using peering address, you should probably stop, clear the data and dir and start fresh as some core configurations could persist and prevent a reset.

Chetan

06:27 AM

clear all data? or just one of the folders?

06:28

Chetan

06:28 AM

Kishore Nallan

06:29 AM

State is the only directory that mattera. Other two are populated from state.

Chetan

06:31 AM

didn't seem to resolve everything - at this point i'm just starting an empty server

Kishore Nallan

06:40 AM

Can you post your full config file?

Chetan

03:23 PM

Kishore Nallan

03:24 PM

Docker networking is the other variable here.

Chetan

03:27 PM

yup, and VM networking. but i've verified those are working because i can ping port 8107 from one VM to the other and get back results that seem correct

03:28

Chetan

03:28 PM

here's a curl from one node to port :8107 of the others (and itself)

08:23

Chetan

08:23 PM

just verified the curl commands work inside the container too

10:06

Chetan

10:06 PM

so they do seem to be talking to each other (i think), it's just the one node talks to another saying "X command is for node 10.0.0.5:8107" and the server responds "yo idk who 10.0.0.5:8107 is, i'm 172.17.0.2:8107:443" (which is a local IP - they're all that specific port funny enough).

Image 1 for so they do seem to be talking to each other (i think), it's just the one node talks to another saying "X command is for node 10.0.0.5:8107" and the server responds "yo idk who 10.0.0.5:8107 is, i'm 172.17.0.2:8107:443" (which is a local IP - they're all that specific port funny enough).

Aug 24, 2023 (1 month ago)

Chetan

04:27 AM

hi Kishore Nallan - any chance the above screenshot gives any insights on what may be up here?

Kishore Nallan

04:28 AM

Can you post the contents of the nodes file also

04:31

Kishore Nallan

04:31 AM

Nodes file should not contain 0.0.0.0 -- it should have internal IPs of the nodes in the cluster.

04:33

Kishore Nallan

04:33 AM

The other suggestion I have is to specify a peering address but given the internal IP. In the screenshots, I see both 10. and 172. getting used so I wonder if there are two networks and Typesense is not picking the correct IP for cluster peering.

04:33

Kishore Nallan

04:33 AM

So give a --peering-address=10.x.x.x explicitly.

Chetan

04:47 AM

here's the current file - just trying to get the node to identify itself as the leader.

04:48

Chetan

04:48 AM

and yes it doesn't seem to recognize 10.0.0.4 as itself

Kishore Nallan

04:48 AM

Try using the 172.x address

04:49

Kishore Nallan

04:49 AM

Also delete data directory and start fresh

Chetan

04:51 AM

the 172.x address is the same though across my VMs - all of them seem to self-identify is 172.17.0.2:8107

04:51

Chetan

04:51 AM

telling them all --peering-address 172.17.0.2:8107 seems wrong?

04:52

Chetan

04:52 AM

here's what happens when i try telling it its IP is 10.0.0.5 though

Kishore Nallan

04:52 AM

Hmm, sorry I am not familiar with Docker networking enough to help further here.

04:53

Kishore Nallan

04:53 AM

Does ifconfig list that 10.x IP?

Chetan

04:58 AM

ah no worries, i didn't realize that docker was causing the failure to listen to 10.0.0.5

04:58

Chetan

04:58 AM

i assumed it was typesense itself.

04:59

Chetan

04:59 AM

the fix for docker was simple - i just added a --network host argument (not the most security-ideal solution, but works for this case)

Kishore Nallan

05:01 AM

👍

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Troubleshooting Multi-Node Setup in Docker

Harpreet experienced issues running a multi-node setup on Docker and received troubleshooting advice and alternative solution from Kishore Nallan.

29mo

Setting Up HA Cluster on Fly with Typesense

Jordan needed help setting up an HA cluster on Fly for Typesense. Jason assisted in troubleshooting the configuration and 6tunnel usage. Eventually, a solution was found in the Fly community forum.

4mo

Troubleshooting Typesense Cluster Multi-node Leadership Error

Bill experienced a problem with a new typesense cluster, receiving an error about no leader and health status issues. Jason and Kishore Nallan provided troubleshooting steps and determined it was likely due to a communication issue between nodes. Kishore Nallan identified a potential solution involving resetting the data directory. Following this, Bill reported the error resolved.

22mo

Testing High Availability with Raft Returns Crashes

pboros reports an issue with usual crashes when testing high availability with Raft. Kishore Nallan suggests checking the quorum recovery period and efficiently logging the crash on all nodes. The issue persists, with pboros suspecting it's due to hostname being no longer resolvable once a container is killed.

15mo

Debugging and Recovery of a Stuck Typesense Cluster

Charlie had a wedged staging cluster. Jason provided debugging and recovery steps, and Adrian helped with more insights. It turns out the issue was insufficient disk space. Once Adrian increased the disk size, the cluster healed itself.

2mo