Hello, we just set up a new typesense cluster and we receive -> Multi-node with no leader: refusing to reset peers.. We hadn't any issue before. All health status in nodes are false and on debug state:4. Any solution?

@Bill If you have at least 3 nodes, then it's most likely that one of the nodes is not able to communicate with one or more nodes. So it's not able to get a quorum to elect a leader in the cluster

@Jason Bosco the steps that I followed are: 1) wget package , 2) install , 3) stop server , 4) create nodes file in etc/typesense , 5) update typesense-server.ini with nodes, peering up etc. 6) start the nodes one by one

Can you post the actual logs from any one node? It seems like a firewall issue to me. Maybe try telnet to the host and port from one of the nodes to the other nodes.

W20220117 111253.172574 2229 raft_server.cpp:551] Multi-node with no leader: refusing to reset peers. I20220117 111254.934175 2240 node.cpp:1484] node default_group10.114.0.48107:8108 term 2 start pre_vote W20220117 111254.934615 2240 node.cpp:1494] node default_group10.114.0.48107:8108 can't do pre_vote as it is not in 10.19.0.781078108 I20220117 111300.795843 2239 node.cpp:1484] node default_group10.114.0.48107:8108 term 2 start pre_vote W20220117 111300.796419 2239 node.cpp:1494] node default_group10.114.0.48107:8108 can't do pre_vote as it is not in 10.19.0.781078108 I20220117 111303.173813 2229 raft_server.cpp:524] Term: 2, last_index index: 1, committed_index: 0, known_applied_index: 0, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 0 W20220117 111303.173848 2229 raft_server.cpp:551] Multi-node with no leader: refusing to reset peers. I20220117 111306.508644 2237 node.cpp:1484] node default_group10.114.0.48107:8108 term 2 start pre_vote W20220117 111306.508692 2237 node.cpp:1494] node default_group10.114.0.48107:8108 can't do pre_vote as it is not in 10.19.0.781078108

node default_group10.114.0.48107:8108 can't do pre_vote as it is not in 10.19.0.781078108

Looks like the node was part of another cluster earlier? I see two different IPs there from entirely different subnets as well.

Yes me too

Can you stop the nodes, delete the data directory and start one by one again?

I have try many times with different order. I stopped 1 node to stabilize like a single node and then I updated the typesense-server.ini in order to work like a multinode

Can you post the contents of the ini file?

I get the same error again _> Multi-node with no leader: refusing to reset peers

Ok that's fine, content of

/etc/typesense/nodes.txt

?

Like the line you have on docs

What is this IP:

10.19.0.7

-> is it some other node on your infrastructure.

10.114.0.281078108,10.114.0.381078108,10.114.0.481078108

If you clear the data directory and restart, there is no way for that IP to come into the picture...

I don't have any node with this ip on digital ocean

Ok, let's try this: 1. Stop Typesense server on all nodes 2. Do

rm -rf /var/lib/typesense/*

on all nodes. 3. Start any one node and check the logs. What do they say?

start the node as single or multi?

Just start it as it is with the multi node configuration.

ok just a moment

Yeah you are good to go then. I'm sure that the data directory had some previous state.

Maybe but this a was an install on a new instance

It deletes the contents of the typesense data directory.

Is it safe to use it with records on production?

You mean whether the cluster is fine to use on production now? Yes, absolutely. I have no idea what happened earlier, but this is clean state.

Yes for example, I have a 3 node cluster now and I want to add 2 more nodes (5 node cluster). If I get this error (Multi-node with no leader: refusing to reset peers) can I use this command?

This will only happen when you mess up the cluster state by replacing all nodes with different IPs at once.

What do you mean from scratch? Set the nodes.txt with 5 peering IPS even if they don't exist now?

I mean, you said that you faced this issue on a brand new droplet right. If you want you can try creating a cluster from new droplets again. But whatever happened earlier, cluster is fine now so should be fine.

Perfect okay, thank you Kishore!

Welcome.