Troubleshooting Multi-Node Setup in Docker
TLDR Harpreet experienced issues running a multi-node setup on Docker and received troubleshooting advice and alternative solution from Kishore Nallan.
2
Jul 09, 2021 (29 months ago)
Harpreet
10:58 AMFirst I did
echo '127.0.0.1:8107:8108,127.0.0.1:7107:7108,127.0.0.1:9107:9108' > `pwd`/typesense-server-nodes`
following [this](https://github.com/typesense/typesense-js/blob/master/package.json#L71).
Then I opened 3 terminals and pasted:
docker run -i -p 8108:8108 -p 8107:8107 -v/tmp/typesense-server-data-1b/:/data -v`pwd`/typesense-server-nodes:/typesense-server-nodes typesense/typesense:0.20.0 --data-dir /data --api-key=xyz --listen-port 8108 --peering-port 8107 --enable-cors --nodes=/typesense-server-nodes```
docker run -i -p 7108:7108 -p 7107:7107 -v/tmp/.typesense-server-data-2b/:/data -vpwd
/typesense-server-nodes:/typesense-server-nodes typesense/typesense:0.20.0 --data-dir /data --api-key=xyz --listen-port 7108 --peering-port 7107 --enable-cors --nodes=/typesense-server-nodes
docker run -i -p 9108:9108 -p 9107:9107 -v/tmp/.typesense-server-data-3b/:/data -vpwd
/typesense-server-nodes:/typesense-server-nodes typesense/typesense:0.20.0 --data-dir /data --api-key=xyz --listen-port 9108 --peering-port 9107 --enable-cors --nodes=/typesense-server-nodes
Log was repeating `Multi-node with no leader: refusing to reset peers.`
So I tried initiating voting process, which returned `success: true` but nothing happened.
All other request are returning `{ "message": "Not Ready or Lagging"}`
What am I missing?
I20210709 10:33:43.281678 129 typesense_server_utils.cpp:260] Typesense peering service is running on 10.0.2.100:8107I20210709 10:33:43.281688 129 typesense_server_utils.cpp:261] Snapshot interval configured as: 3600s
W20210709 10:33:43.281697 129 controller.cpp:1454] SIGINT was installed with 1
I20210709 10:33:43.281752 129 raft_server.cpp:511] Term: 1, last_index index: 0, committed_index: 0, known_applied_index: 0, applying_index: 0, pending_index: 0, disk_index: 0, pending_queue_size: 0, local_sequence: 0
W20210709 10:33:43.281774 129 raft_server.cpp:539] Multi-node with no leader: refusing to reset peers.
I20210709 10:33:53.283394 129 raft_server.cpp:511] Term: 1, last_index index: 0, committed_index: 0, known_applied_index: 0, applying_index: 0, pending_index: 0, disk_index: 0, pending_queue_size: 0, local_sequence: 0
W20210709 10:33:53.283416 129 raft_server.cpp:539] Multi-node with no leader: refusing to reset peers.
I20210709 10:33:54.136162 138 node.cpp:1484] node default_group:10.0.2.100:8107:8108 term 1 start pre_vote
W20210709 10:33:54.136191 138 node.cpp:1494] node default_group:10.0.2.100:8107:8108 can't do pre_vote as it is not in 127.0.0.1:7107:7108,127.0.0.1:8107:8108,127.0.0.1:9107:9108
I20210709 10:34:00.097368 139 node.cpp:1484] node default_group:10.0.2.100:8107:8108 term 1 start pre_vote
W20210709 10:34:00.097395 139 node.cpp:1494] node default_group:10.0.2.100:8107:8108 can't do pre_vote as it is not in 127.0.0.1:7107:7108,127.0.0.1:8107:8108,127.0.0.1:9107:9108
I20210709 10:34:02.366971 43 node.cpp:1211] node default_group:10.0.2.100:8107:8108 trigger-vote, current_term 1 state FOLLOWER election_timeout 5000
I20210709 10:34:02.366998 43 raft_server.cpp:664] Triggered vote. Ok? 1, status: OK
I20210709 10:34:03.284966 129 raft_server.cpp:511] Term: 1, last_index index: 0, committed_index: 0, known_applied_index: 0, applying_index: 0, pending_index: 0, disk_index: 0, pending_queue_size: 0, local_sequence: 0
W20210709 10:34:03.284992 129 raft_server.cpp:539] Multi-node with no leader: refusing to reset peers.
I20210709 10:34:08.323179 138 node.cpp:1484] node default_group:10.0.2.100:8107:8108 term 1 start pre_vote
W20210709 10:34:08.323215 138 node.cpp:1494] node default_group:10.0.2.100:8107:8108 can't do pre_vote as it is not in 127.0.0.1:7107:7108,127.0.0.1:8107:8108,127.0.0.1:9107:9108```
Kishore Nallan
11:00 AMHarpreet
11:02 AMKishore Nallan
11:02 AMHarpreet
11:02 AMKishore Nallan
11:03 AM1
Kishore Nallan
11:03 AMHarpreet
11:07 AMKishore Nallan
11:08 AMHarpreet
11:09 AMKishore Nallan
11:09 AM1
Kishore Nallan
11:09 AMHarpreet
11:10 AMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Troubleshooting Typesense Cluster Multi-node Leadership Error
Bill experienced a problem with a new typesense cluster, receiving an error about no leader and health status issues. Jason and Kishore Nallan provided troubleshooting steps and determined it was likely due to a communication issue between nodes. Kishore Nallan identified a potential solution involving resetting the data directory. Following this, Bill reported the error resolved.
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
Troubleshooting IP Update on Kubernetes Typesense
Alessandro and Damien are having issues with old IP addresses in a Kubernetes Typesense cluster not being updated. Kishore Nallan provides possible troubleshooting solutions, and mentioned the need for a fix for DNS retries. A suggested update strategy was shared by Aljosa.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.