We have a cluster completely down for unknown reas...
# community-help
ó
We have a cluster completely down for unknown reasons. There's nothing on the logs. Once it finishes loading all the collections it just restarts (30.1). It takes over an hour to restart with 5M+ documents and 40Gb+ of ram. skip-writes didn't help on the .ini file didn't help. Here's the log of the restart:
Copy code
I20260228 10:16:28.447320  1350 collection_manager.cpp:2101] Loaded 4718592 documents from tenders-documents so far.
I20260228 10:16:59.182713  1241 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 0, reference_q.size: 0
Log file created at: 2026/02/28 10:17:02
Running on machine: typesense-1
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20260228 10:17:02.948997 10630 typesense_server_utils.cpp:480] Starting Typesense 30.1
I20260228 10:17:02.949477 10630 typesense_server_utils.cpp:483] Typesense is using jemalloc.
I20260228 10:17:02.949595 10630 typesense_server_utils.cpp:550] Thread pool size: 64
I20260228 10:17:02.957405 10630 store.cpp:39] Initializing DB by opening state dir: /data/typesense/db
I20260228 10:17:05.189091 10630 store.cpp:39] Initializing DB by opening state dir: /data/typesense/meta
I20260228 10:17:05.218000 10630 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20260228 10:17:05.218021 10630 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20260228 10:17:05.231212 10630 typesense_server_utils.cpp:664] Loaded 0 natural language search model(s).
I20260228 10:17:05.231293 10630 typesense_server_utils.cpp:736] Starting API service...
I20260228 10:17:05.231472 10890 batched_indexer.cpp:196] Starting batch indexer with 64 threads.
I20260228 10:17:05.231514 10892 typesense_server_utils.cpp:679] Conversation garbage collector thread started.
I20260228 10:17:05.231936 10630 http_server.cpp:243] Typesense has started listening on port 8108
I20260228 10:17:05.233827 10890 batched_indexer.cpp:201] BatchedIndexer skip_index: -9999
I20260228 10:17:05.234226 10968 batched_indexer.cpp:353] Starting reference sequence thread.
I20260228 10:17:05.236889 10889 server.cpp:1181] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20260228 10:17:05.236908 10889 server.cpp:1184] Check out http://typesense-1:8107 in web browser.
I20260228 10:17:05.245069 10889 raft_server.cpp:79] Nodes configuration: 10.0.1.4:8107:8108,10.0.1.13:8107:8108,10.0.1.12:8107:8108
I20260228 10:17:05.245631 10889 log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I20260228 10:17:05.245702 10889 log.cpp:1172] log load_meta /data/typesense/state/log/log_meta first_log_index: 1885071 time: 42
I20260228 10:17:05.245725 10889 log.cpp:1112] load open segment, path: /data/typesense/state/log first_index: 1885067
I20260228 10:17:05.259920 10941 raft_server.cpp:710] on_snapshot_load
I20260228 10:17:05.346503 10941 store.cpp:245] rm /data/typesense/db success
I20260228 10:17:05.464650 10941 store.cpp:255] copy snapshot /data/typesense/state/snapshot/snapshot_00000000000001885073/db_snapshot to /data/typesense/db success
I20260228 10:17:05.464713 10941 store.cpp:39] Initializing DB by opening state dir: /data/typesense/db
I20260228 10:17:05.610216 10941 store.cpp:269] DB open success!
I20260228 10:17:05.610250 10941 raft_server.cpp:664] Loading collections from disk...
I20260228 10:17:05.610265 10941 collection_manager.cpp:426] CollectionManager::load()
I20260228 10:17:05.611490 10941 auth_manager.cpp:35] Indexing 0 API key(s) found on disk.
I20260228 10:17:05.635917 10941 collection_manager.cpp:456] Loading symlink favorites-collection to favorites-documents
I20260228 10:17:05.635947 10941 collection_manager.cpp:456] Loading symlink tenders-collection to tenders-documents
I20260228 10:17:05.635959 10941 collection_manager.cpp:462] Loading upto 32 collections in parallel, 20000 documents at a time.
I20260228 10:17:05.668130 10941 collection_manager.cpp:471] Found 702 collection(s) on disk.
Is there anything else we can look at?
One machine seems to boot, but it never becomes the leader, so it doesn't work at all.
Copy code
I20260228 09:57:25.073647  1331 collection_manager.cpp:2101] Loaded 4718592 documents from tenders-documents so far.
I20260228 09:58:02.445748  1175 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 0, reference_q.size: 0
I20260228 09:58:06.599809  1331 collection_manager.cpp:2101] Loaded 4751360 documents from tenders-documents so far.
I20260228 09:58:33.441555  1331 collection_manager.cpp:2112] Indexed 4784207/4784207 documents into collection tenders-documents
I20260228 09:58:33.463333  1240 collection_manager.cpp:628] Loaded 714 collection(s).
I20260228 09:58:33.463362  1240 collection_manager.cpp:629] Found 0 analytics config(s).
I20260228 09:58:33.463369  1240 collection_manager.cpp:634] Loaded 0 analytics config(s).
I20260228 09:58:33.464090  1240 collection_manager.cpp:641] Found 0 old analytics config(s) on disk.
I20260228 09:58:33.464107  1240 collection_manager.cpp:652] Restored 0 old analytics config(s). from 0 old analytics config(s) on disk.
I20260228 09:58:33.464114  1240 collection_manager.cpp:653] Removing restored old analytics config(s) from disk.
I20260228 09:58:33.466796  1240 raft_server.cpp:671] Finished loading collections from disk.
I20260228 09:58:33.466892  1240 raft_server.cpp:682] Loaded 0 conversation model(s).
I20260228 09:58:33.466902  1240 raft_server.cpp:686] Initializing batched indexer from snapshot state...
I20260228 09:58:33.466956  1240 batched_indexer.cpp:632] Restored 0 in-flight requests from snapshot.
I20260228 09:58:33.467212  1240 raft_server.cpp:699] Loaded 0 personalization model(s).
I20260228 09:58:33.468297  1240 raft_server.h:304] Configuration of this group is 10.0.1.4:8107:8108,10.0.1.12:8107:8108,10.0.1.13:8107:8108
I20260228 09:58:33.473151  1240 snapshot_executor.cpp:264] node default_group:10.0.1.4:8107:8108 snapshot_load_done, last_included_index: 1885472 last_included_term: 64 peers: "10.0.1.4:8107:8108" peers: "10.0.1.12:8107:8108" peers: "10.0.1.13:8107:8108"
I20260228 09:58:33.475670  1174 raft_meta.cpp:521] Loaded single stable meta, path /data/typesense/state/meta term 64 votedfor 10.0.1.4:8107:8108 time: 331
I20260228 09:58:33.476011  1174 node.cpp:608] node default_group:10.0.1.4:8107:8108 init, term: 64 last_log_id: (index=1885472,term=64) conf: 10.0.1.4:8107:8108,10.0.1.12:8107:8108,10.0.1.13:8107:8108 old_conf:
I20260228 09:58:33.476045  1174 raft_server.cpp:145] Node last_index: 1885472
I20260228 09:58:33.476059  1174 typesense_server_utils.cpp:424] Typesense peering service is running on 10.0.1.4:8107
I20260228 09:58:33.476068  1174 typesense_server_utils.cpp:425] Snapshot interval configured as: 3600s
I20260228 09:58:33.476074  1174 typesense_server_utils.cpp:426] Snapshot max byte count configured as: 4194304
W20260228 09:58:33.476239  1174 controller.cpp:1550] SIGINT was installed with 1
I20260228 09:58:33.476301  1174 raft_server.cpp:758] Term: 64, pending_queue: 0, last_index: 1885472, committed: 0, known_applied: 1885472, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
W20260228 09:58:33.476310  1174 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 09:58:33.476320  1174 raft_server.cpp:1130] Snapshot timer is active, current_ts: 1772272713, last_snapshot_ts: 1772267862
I20260228 09:58:33.476328  1174 node.cpp:943] node default_group:10.0.1.4:8107:8108 starts to do snapshot
I20260228 09:58:33.476349  1174 snapshot_executor.cpp:154] node default_group:10.0.1.4:8107:8108 the gap between fsm applied index 1885472 and last_snapshot_index 1885472 is less than 1, will clear bufferred logs and return success
I20260228 09:58:33.476357  1240 raft_server.cpp:1214] Timed snapshot succeeded!
I20260228 09:58:39.304322  1240 node.cpp:1579] node default_group:10.0.1.4:8107:8108 term 64 start pre_vote
W20260228 09:58:39.312534  1244 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.12:8107:8108 error: [E2][10.0.1.12:8107][E2]peer_id not exist
W20260228 09:58:39.312808  1240 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.13:8107:8108 error: [E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 09:58:43.477579  1174 raft_server.cpp:758] Term: 64, pending_queue: 0, last_index: 1885472, committed: 0, known_applied: 1885472, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
W20260228 09:58:43.477636  1174 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 09:58:45.303790  1240 node.cpp:1579] node default_group:10.0.1.4:8107:8108 term 64 start pre_vote
W20260228 09:58:45.306088  1246 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.12:8107:8108 error: [E2][10.0.1.12:8107][E2]peer_id not exist
W20260228 09:58:45.306699  1250 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.13:8107:8108 error: [E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 09:58:50.426270  1240 node.cpp:1579] node default_group:10.0.1.4:8107:8108 term 64 start pre_vote
W20260228 09:58:50.428284  1244 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.12:8107:8108 error: [E2][10.0.1.12:8107][E2]peer_id not exist
W20260228 09:58:50.428980  1240 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.13:8107:8108 error: [E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 09:58:53.479035  1174 raft_server.cpp:758] Term: 64, pending_queue: 0, last_index: 1885472, committed: 0, known_applied: 1885472, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
W20260228 09:58:53.479085  1174 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 09:58:56.124789  1240 node.cpp:1579] node default_group:10.0.1.4:8107:8108 term 64 start pre_vote
W20260228 09:58:56.126942  1244 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.12:8107:8108 error: [E2][10.0.1.12:8107][E2]peer_id not exist
W20260228 09:58:56.127346  1240 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.13:8107:8108 error: [E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 09:59:01.486280  1240 node.cpp:1579] node default_group:10.0.1.4:8107:8108 term 64 start pre_vote
W20260228 09:59:01.488619  1244 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.12:8107:8108 error: [E2][10.0.1.12:8107][E2]peer_id not exist
W20260228 09:59:01.488701  1240 node.cpp:1559] node default_group:10.0.1.4:8107:8108 request PreVote from 10.0.1.13:8107:8108 error: [E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 09:59:03.454255  1175 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 0, reference_q.size: 0
I20260228 09:59:03.480365  1174 raft_server.cpp:758] Term: 64, pending_queue: 0, last_index: 1885472, committed: 0, known_applied: 1885472, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
W20260228 09:59:03.480392  1174 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
stuck with Multi-node with no leader: refusing to reset peers. all the time, returning 503 not ready or lagging
Trying this approach at https://typesense.org/docs/guide/high-availability.html#when-to-use-snapshot-restore, the first node becomes leader and starts working, but once I add a second one it never synchronice:
Copy code
I20260228 10:40:50.774179 13210 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:40:50.776796 13200 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:40:53.501308 13168 raft_server.cpp:758] Term: 66, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260228 10:40:53.501358 13168 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 10:40:56.212661 13200 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:40:56.215327 13210 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:01.827168 13210 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:01.829367 13200 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:03.502465 13168 raft_server.cpp:758] Term: 66, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260228 10:41:03.502508 13168 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 10:41:06.877609 13200 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:06.880528 13210 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:11.917071 13210 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:11.919591 13200 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:13.503639 13168 raft_server.cpp:758] Term: 66, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260228 10:41:13.503712 13168 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 10:41:17.136611 13200 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:17.139066 13210 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:17.463721 13169 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 0, reference_q.size: 0
I20260228 10:41:22.879053 13210 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:22.881788 13200 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:23.504770 13168 raft_server.cpp:758] Term: 66, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260228 10:41:23.504809 13168 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 10:41:28.760584 13200 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:28.763582 13210 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
I20260228 10:41:33.505930 13168 raft_server.cpp:758] Term: 66, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 0
W20260228 10:41:33.505977 13168 raft_server.cpp:786] Multi-node with no leader: refusing to reset peers.
I20260228 10:41:33.991089 13210 node.cpp:1579] node default_group:10.0.1.13:8107:8108 term 66 start pre_vote
I20260228 10:41:33.993561 13200 node.cpp:1504] node default_group:10.0.1.13:8107:8108 received PreVoteResponse from 10.0.1.4:8107:8108 term 66 granted 0 rejected_by_lease 0 disrupted 1
Now we get a failed to catch up error
Copy code
I20260228 12:32:25.066777  1240 replicator.cpp:1252] Replicator=2052994367561 is going to quit
E20260228 12:32:25.066883  1240 raft_server.h:62] Peer refresh failed, error: Peer 10.0.1.13:8107:8108 failed to catch up
I20260228 12:32:25.067431  1174 raft_server.cpp:758] Term: 68, pending_queue: 0, last_index: 1885474, committed: 1885474, known_applied: 1885474, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
I20260228 12:32:25.067456  1174 node.cpp:3189] node default_group:10.0.1.4:8107:8108 change_peers from 10.0.1.4:8107:8108 to 10.0.1.4:8107:8108,10.0.1.13:8107:8108, begin caughtup.
I20260228 12:32:25.067481  1174 replicator.cpp:143] Replicator=4385161609281@10.0.1.13:8107:8108 is started, group default_group
W20260228 12:32:25.071349  1242 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=1, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
W20260228 12:32:27.583648  1240 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=11, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 12:32:30.067807  1244 node.cpp:754] node default_group:10.0.1.4:8107:8108 waits peer 10.0.1.13:8107:8108 to catch up
W20260228 12:32:30.097289  1240 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=21, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
W20260228 12:32:32.612375  1246 replicator.cpp:297] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=31, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
W20260228 12:32:35.068118  1244 node.cpp:3245] Node default_group:10.0.1.4:8107:8108 fail to catch up peer 10.0.1.13:8107:8108 when trying to change peers from 10.0.1.4:8107:8108 to 10.0.1.4:8107:8108,10.0.1.13:8107:8108
I20260228 12:32:35.068192  1244 node.cpp:3298] node default_group:10.0.1.4:8107:8108 reset ConfigurationCtx, new_peers: 10.0.1.4:8107:8108,10.0.1.13:8107:8108, old_peers: 10.0.1.4:8107:8108
I20260228 12:32:35.068209  1244 replicator.cpp:988] Group default_group Replicator=4385161609281 is going to quit
I20260228 12:32:35.068218  1244 replicator.cpp:1252] Replicator=4385161609281 is going to quit
E20260228 12:32:35.068327  1244 raft_server.h:62] Peer refresh failed, error: Peer 10.0.1.13:8107:8108 failed to catch up
I20260228 12:32:35.068949  1174 raft_server.cpp:758] Term: 68, pending_queue: 0, last_index: 1885474, committed: 1885474, known_applied: 1885474, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
I20260228 12:32:35.068975  1174 node.cpp:3189] node default_group:10.0.1.4:8107:8108 change_peers from 10.0.1.4:8107:8108 to 10.0.1.4:8107:8108,10.0.1.13:8107:8108, begin caughtup.
I20260228 12:32:35.069003  1174 replicator.cpp:143] Replicator=3483218477121@10.0.1.13:8107:8108 is started, group default_group
I20260228 12:32:35.069124  1174 raft_server.cpp:1130] Snapshot timer is active, current_ts: 1772281955, last_snapshot_ts: 1772272713
W20260228 12:32:35.071339  1174 raft_server.cpp:1167] Peer 10.0.1.13:8107:8108 reported unhealthy during snapshot pre-check.
W20260228 12:32:35.071355  1174 raft_server.cpp:1174] Unable to trigger snapshot as one or more of the peers reported unhealthy.
W20260228 12:32:35.073448  1242 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=1, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 12:32:35.566843  1175 batched_indexer.cpp:432] Running GC for aborted requests, req map size: 0, reference_q.size: 0
W20260228 12:32:37.589491  1242 replicator.cpp:297] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=11, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
I20260228 12:32:40.069352  1244 node.cpp:754] node default_group:10.0.1.4:8107:8108 waits peer 10.0.1.13:8107:8108 to catch up
W20260228 12:32:40.106173  1249 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=21, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
W20260228 12:32:42.623054  1249 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=31, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
W20260228 12:32:45.069660  1244 node.cpp:3245] Node default_group:10.0.1.4:8107:8108 fail to catch up peer 10.0.1.13:8107:8108 when trying to change peers from 10.0.1.4:8107:8108 to 10.0.1.4:8107:8108,10.0.1.13:8107:8108
I20260228 12:32:45.069738  1244 node.cpp:3298] node default_group:10.0.1.4:8107:8108 reset ConfigurationCtx, new_peers: 10.0.1.4:8107:8108,10.0.1.13:8107:8108, old_peers: 10.0.1.4:8107:8108
I20260228 12:32:45.069759  1244 replicator.cpp:988] Group default_group Replicator=3483218477121 is going to quit
I20260228 12:32:45.069767  1244 replicator.cpp:1252] Replicator=3483218477121 is going to quit
E20260228 12:32:45.069870  1244 raft_server.h:62] Peer refresh failed, error: Peer 10.0.1.13:8107:8108 failed to catch up
I20260228 12:32:45.072781  1174 raft_server.cpp:758] Term: 68, pending_queue: 0, last_index: 1885474, committed: 1885474, known_applied: 1885474, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
I20260228 12:32:45.072809  1174 node.cpp:3189] node default_group:10.0.1.4:8107:8108 change_peers from 10.0.1.4:8107:8108 to 10.0.1.4:8107:8108,10.0.1.13:8107:8108, begin caughtup.
I20260228 12:32:45.072835  1174 replicator.cpp:143] Replicator=6721623818257@10.0.1.13:8107:8108 is started, group default_group
W20260228 12:32:45.076890  1242 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=1, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
W20260228 12:32:47.589036  1250 replicator.cpp:397] Group default_group fail to issue RPC to 10.0.1.13:8107:8108 _consecutive_error_times=11, [E2][10.0.1.13:8107][E2]peer_id not exist [R1][E2][10.0.1.13:8107][E2]peer_id not exist [R2][E2][10.0.1.13:8107][E2]peer_id not exist [R3][E2][10.0.1.13:8107][E2]peer_id not exist
Pretty similar log on the second node
Copy code
I20260228 12:22:25.567296 15197 collection_manager.cpp:2101] Loaded 4734976 documents from tenders-documents so far.
E20260228 12:22:32.624740 13168 raft_server.cpp:814] Node not ready yet (known_applied_index is 0).
I20260228 12:22:34.625149 13168 raft_server.cpp:758] Term: 68, pending_queue: 0, last_index: 0, committed: 0, known_applied: 0, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 120539425
E20260228 12:22:41.626000 13168 raft_server.cpp:814] Node not ready yet (known_applied_index is 0).
Log file created at: 2026/02/28 12:22:45
Running on machine: typesense-1
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20260228 12:22:45.969138 23191 typesense_server_utils.cpp:480] Starting Typesense 30.1
I20260228 12:22:45.970069 23191 typesense_server_utils.cpp:483] Typesense is using jemalloc.
I20260228 12:22:45.970212 23191 typesense_server_utils.cpp:550] Thread pool size: 64
I20260228 12:22:45.977594 23191 store.cpp:39] Initializing DB by opening state dir: /data/typesense/db
I20260228 12:22:48.324342 23191 store.cpp:39] Initializing DB by opening state dir: /data/typesense/meta
j
Hi Oscar, is the cluster hosted on Typesense Cloud? If so I can take a look if you post or message me the cluster ID.
ó
Nop, self hosted. We managed to fix it. For some reason it was an oom even if there were more than 20Gb free of ram. We added more Swap and somehow it fixed it. We still don't know why it happens 🤷‍♂️