Hi, I'm facing an issue with some nodes in my sel...
# community-help
o
Hi, I'm facing an issue with some nodes in my self hosted clusters not joining their cluster properly with "reject term_unmatched AppendEntries" errors that I must admit I don't know how to interpret:
Copy code
I20260305 08:51:27.114995   212 node.cpp:3298] node default_group:192.168.0.42:8107:8108 reset ConfigurationCtx, new_peers: 192.168.0.42:8107:8108, old_peers: 192.168.0.42:8107:8108
I20260305 08:51:27.115005   212 raft_server.h:277] Node becomes leader, term: 409
W20260305 08:51:27.130923   207 node.cpp:2390] node default_group:192.168.0.42:8107:8108 ignore stale AppendEntries from 192.168.3.132:8107:8108 in term 408 current_term 409
W20260305 08:51:27.130928   210 node.cpp:2390] node default_group:192.168.0.42:8107:8108 ignore stale AppendEntries from 192.168.3.132:8107:8108 in term 408 current_term 409
time=2026-03-05T08:51:29.563Z level=INFO msg="collecting data" cluster=vnext-typesense-operator endpoint=http://localhost:8108
time=2026-03-05T08:51:29.563Z level=INFO msg="collecting metrics" cluster=vnext-typesense-operator url=http://localhost:8108/metrics.json
time=2026-03-05T08:51:29.670Z level=INFO msg="collecting stats" cluster=vnext-typesense-operator url=http://localhost:8108/stats.json
time=2026-03-05T08:51:29.670Z level=INFO msg="collecting data completed" duration=106.85736ms
I20260305 08:51:32.740042   211 node.cpp:2120] node default_group:192.168.0.42:8107:8108 received PreVote from 192.168.3.132:8107:8108 in term 410 current_term 409 granted 0 rejected_by_lease 0
I20260305 08:51:32.747063   212 node.cpp:2202] node default_group:192.168.0.42:8107:8108 received RequestVote from 192.168.3.132:8107:8108 in term 410 current_term 409 log_is_ok 0 votable_time 0
I20260305 08:51:32.747112   210 raft_server.h:282] Node stepped down : Raft node receives higher term request_vote_request.
I20260305 08:51:32.784587   212 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 410 votedfor 0.0.0.0:0:0 time: 37472
W20260305 08:51:32.841040   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7485 prev_log_term 404 local_prev_log_term 409 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.842417   212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7484 prev_log_term 402 local_prev_log_term 407 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.843758   210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7483 prev_log_term 402 local_prev_log_term 405 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.845094   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7482 prev_log_term 400 local_prev_log_term 403 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.846421   212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7481 prev_log_term 398 local_prev_log_term 401 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.847744   210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7480 prev_log_term 396 local_prev_log_term 399 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
I20260305 08:51:32.848554   211 raft_server.h:299] Node starts following { leader_id=192.168.3.132:8107:8108, term=410, status=Follower receives message from new leader with the same term.}
W20260305 08:51:32.849092   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7479 prev_log_term 394 local_prev_log_term 397 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.850427   212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7478 prev_log_term 392 local_prev_log_term 395 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.851759   210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7477 prev_log_term 390 local_prev_log_term 393 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.853088   211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7476 prev_log_term 388 local_prev_log_term 391 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.854426   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7475 prev_log_term 386 local_prev_log_term 389 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.855820   212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7474 prev_log_term 384 local_prev_log_term 387 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.857210   210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7473 prev_log_term 382 local_prev_log_term 385 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.859202   211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7472 prev_log_term 378 local_prev_log_term 383 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.860495   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7471 prev_log_term 376 local_prev_log_term 379 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.861850   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7470 prev_log_term 376 local_prev_log_term 377 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.863174   210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7469 prev_log_term 374 local_prev_log_term 375 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.864518   211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7468 prev_log_term 372 local_prev_log_term 373 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.865862   212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7467 prev_log_term 370 local_prev_log_term 371 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.867173   207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7466 prev_log_term 368 local_prev_log_term 369 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.868505   210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7465 prev_log_term 366 local_prev_log_term 367 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.869843   211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7464 prev_log_term 366 local_prev_log_term 365 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
F20260305 08:51:32.872825   207 log_manager.cpp:310] Can't truncate logs before _applied_id=7485, last_log_kept=7463
Then comes the "terminating abruptly" ;)
Copy code
*** Check failure stack trace: ***
E20260305 08:51:33.874399   207 backward.hpp:4200] Stack trace (most recent call last) in thread 207:
E20260305 08:51:33.874464   207 backward.hpp:4200] #19   Object "/opt/typesense-server", at 0x55cc4265ce40, in bthread_make_fcontext
E20260305 08:51:33.874476   207 backward.hpp:4200] #18   Object "/opt/typesense-server", at 0x55cc4267e2f8, in bthread::TaskGroup::task_runner(long)
E20260305 08:51:33.874485   207 backward.hpp:4200] #17   Object "/opt/typesense-server", at 0x55cc425c6794, in brpc::Socket::ProcessEvent(void*)
E20260305 08:51:33.874500   207 backward.hpp:4200] #16   Object "/opt/typesense-server", at 0x55cc4245920b, in brpc::InputMessenger::OnNewMessages(brpc::Socket*)
E20260305 08:51:33.874507   207 backward.hpp:4200] #15   Object "/opt/typesense-server", at 0x55cc4245831e, in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
E20260305 08:51:33.874516   207 backward.hpp:4200] #14   Object "/opt/typesense-server", at 0x55cc42458222, in brpc::ProcessInputMessage(void*)
E20260305 08:51:33.874523   207 backward.hpp:4200] #13   Object "/opt/typesense-server", at 0x55cc42467853, in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*)
E20260305 08:51:33.874532   207 backward.hpp:4200] #12   Object "/opt/typesense-server", at 0x55cc423fe1fa, in braft::RaftService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*)
E20260305 08:51:33.874540   207 backward.hpp:4200] #11   Object "/opt/typesense-server", at 0x55cc4239b989, in braft::RaftServiceImpl::append_entries(google::protobuf::RpcController*, braft::AppendEntriesRequest const*, braft::AppendEntriesResponse*, google::protobuf::Closure*)
E20260305 08:51:33.874547   207 backward.hpp:4200] #10   Object "/opt/typesense-server", at 0x55cc4231c71d, in braft::NodeImpl::handle_append_entries_request(brpc::Controller*, braft::AppendEntriesRequest const*, braft::AppendEntriesResponse*, google::protobuf::Closure*, bool)
E20260305 08:51:33.874555   207 backward.hpp:4200] #9    Object "/opt/typesense-server", at 0x55cc42343560, in braft::LogManager::append_entries(std::vector<braft::LogEntry*, std::allocatorbraft::LogEntry* >*, braft::LogManager::StableClosure*)
E20260305 08:51:33.874564   207 backward.hpp:4200] #8    Object "/opt/typesense-server", at 0x55cc4234321c, in braft::LogManager::check_and_resolve_conflict(std::vector<braft::LogEntry*, std::allocatorbraft::LogEntry* >*, braft::LogManager::StableClosure*)
E20260305 08:51:33.874572   207 backward.hpp:4200] #7    Object "/opt/typesense-server", at 0x55cc42342b9b, in braft::LogManager::unsafe_truncate_suffix(long)
E20260305 08:51:33.874583   207 backward.hpp:4200] #6    Object "/opt/typesense-server", at 0x55cc428f7529, in google::LogMessageFatal::~LogMessageFatal()
E20260305 08:51:33.874589   207 backward.hpp:4200] #5    Object "/opt/typesense-server", at 0x55cc428f3a5f, in google::LogMessage::Flush()
E20260305 08:51:33.874598   207 backward.hpp:4200] #4    Object "/opt/typesense-server", at 0x55cc428f422d, in google::LogMessage::SendToLog()
E20260305 08:51:33.874605   207 backward.hpp:4200] #3    Object "/opt/typesense-server", at 0x55cc428f42d1, in google::LogMessage::Fail()
E20260305 08:51:33.874616   207 backward.hpp:4200] #2    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb315bd57f2, in abort
E20260305 08:51:33.874624   207 backward.hpp:4200] #1    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb315bef475, in raise
E20260305 08:51:33.874632   207 backward.hpp:4200] #0    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb315c439fc, in pthread_kill
Aborted (Signal sent by tkill() 1 10000)
I20260305 08:51:34.896282   207 housekeeper.cpp:96] No in-flight search queries were found.
E20260305 08:51:34.896332   207 typesense_server.cpp:159] Typesense 29.0 is terminating abruptly.
I'm having this issue with both v29.0 and v30.1.
a
Hey @Olivier seems like the raft logs got corrupted at some point. The best path is completely deleting the data dir from the follower node and letting it receive a fresh snapshot
🙏 1