Hey all, question about Typesense cluster syncing ...
# community-help
w
Hey all, question about Typesense cluster syncing issues. I have a self hosted 5 node typesense cluster and when adding a new node and having the node sync with the cluster, it seems the syncing is incomplete sometimes but not all the time. for instance out of 100k records, i might be missing 100-5000 records randomly. I have some errors from my logs that might be of help. Just trying to see what's the best way to go about preventing this:
Copy code
E20250622 09:30:52.754386  2220 raft_server.h:62] Peer refresh failed, error: Peer 10.62.56.25:8107:8108 failed to catch up (edited) 


W20250622 09:33:14.279778  2217 baidu_rpc_protocol.cpp:278] Fail to write into Socket{id=17179869185 fd=134 addr=10.62.56.116:43560:8107} (0xffff5c971d80): Unknown error 1014 [1014]


W20250622 09:30:57.752395  2213 replicator.cpp:1016] Replicator is destroyed when catch_up_timedout.


W20250622 09:32:27.897570  2220 node.cpp:2462] node default_group:10.62.56.59:8107:8108 reject term_unmatched AppendEntries from 10.62.56.91:8107:8108 in term 9 prev_log_index 2 prev_log_term 9 local_prev_log_term 2 last_log_index 6087 entries_size 0 from_append_entries_cache: 0


W20250622 09:32:28.333285  2217 node.cpp:2462] node default_group:10.62.56.59:8107:8108 reject term_unmatched AppendEntries from 10.62.56.91:8107:8108 in term 9 prev_log_index 4 prev_log_term 9 local_prev_log_term 2 last_log_index 6087 entries_size 0 from_append_entries_cache: 0
k
From the logs it seems like you have networking issues. That could be either due to high load or physical routing errors. It's difficult for us to really answer what could be going wrong in these cases because so many environment specific issues could be at play.
w
Yeah, so taking back your feedback I did a test on a cluster with 0 load, 3 nodes, 4 cpus, 16 gigs of ram each. Got them all synced up and they're working great. As a test I added a 4th node and it joined the cluster properly and everything, I got this log in specific regarding one of my collections:
Copy code
I20250624 19:51:44.581035  3290 collection_manager.cpp:2276] Indexed 11080/11080 documents into collection some_collection
Now this is great, this is the number of documents inside of my collection, but when I search on this node in specific I was seeing that I was missing results again. So i took a look at the node itself and got it's collections and see that it indeed had a lower number of documents than 11080.
Copy code
"name":"some_collection","num_documents":10741,"symbols_to_index":[],"token_separators":[]},
So I think I've crossed off the network as an issue, seeing as the log says that it got 11080 items indexed. So I'm not too sure what exactly is going wrong from here. I'm on version 26 of typesense btw. I'm also attaching the full log but the relevant portion is in line 22.
Also, just curious, what configuration does typesense cloud use typically for each node?
k
Version 26 is a couple of versions behind. Please try upgrade to v28.0