Olivier
03/05/2026, 8:55 AMI20260305 08:51:27.114995 212 node.cpp:3298] node default_group:192.168.0.42:8107:8108 reset ConfigurationCtx, new_peers: 192.168.0.42:8107:8108, old_peers: 192.168.0.42:8107:8108
I20260305 08:51:27.115005 212 raft_server.h:277] Node becomes leader, term: 409
W20260305 08:51:27.130923 207 node.cpp:2390] node default_group:192.168.0.42:8107:8108 ignore stale AppendEntries from 192.168.3.132:8107:8108 in term 408 current_term 409
W20260305 08:51:27.130928 210 node.cpp:2390] node default_group:192.168.0.42:8107:8108 ignore stale AppendEntries from 192.168.3.132:8107:8108 in term 408 current_term 409
time=2026-03-05T08:51:29.563Z level=INFO msg="collecting data" cluster=vnext-typesense-operator endpoint=http://localhost:8108
time=2026-03-05T08:51:29.563Z level=INFO msg="collecting metrics" cluster=vnext-typesense-operator url=http://localhost:8108/metrics.json
time=2026-03-05T08:51:29.670Z level=INFO msg="collecting stats" cluster=vnext-typesense-operator url=http://localhost:8108/stats.json
time=2026-03-05T08:51:29.670Z level=INFO msg="collecting data completed" duration=106.85736ms
I20260305 08:51:32.740042 211 node.cpp:2120] node default_group:192.168.0.42:8107:8108 received PreVote from 192.168.3.132:8107:8108 in term 410 current_term 409 granted 0 rejected_by_lease 0
I20260305 08:51:32.747063 212 node.cpp:2202] node default_group:192.168.0.42:8107:8108 received RequestVote from 192.168.3.132:8107:8108 in term 410 current_term 409 log_is_ok 0 votable_time 0
I20260305 08:51:32.747112 210 raft_server.h:282] Node stepped down : Raft node receives higher term request_vote_request.
I20260305 08:51:32.784587 212 raft_meta.cpp:546] Saved single stable meta, path /usr/share/typesense/data/state/meta term 410 votedfor 0.0.0.0:0:0 time: 37472
W20260305 08:51:32.841040 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7485 prev_log_term 404 local_prev_log_term 409 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.842417 212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7484 prev_log_term 402 local_prev_log_term 407 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.843758 210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7483 prev_log_term 402 local_prev_log_term 405 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.845094 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7482 prev_log_term 400 local_prev_log_term 403 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.846421 212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7481 prev_log_term 398 local_prev_log_term 401 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.847744 210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7480 prev_log_term 396 local_prev_log_term 399 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
I20260305 08:51:32.848554 211 raft_server.h:299] Node starts following { leader_id=192.168.3.132:8107:8108, term=410, status=Follower receives message from new leader with the same term.}
W20260305 08:51:32.849092 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7479 prev_log_term 394 local_prev_log_term 397 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.850427 212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7478 prev_log_term 392 local_prev_log_term 395 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.851759 210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7477 prev_log_term 390 local_prev_log_term 393 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.853088 211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7476 prev_log_term 388 local_prev_log_term 391 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.854426 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7475 prev_log_term 386 local_prev_log_term 389 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.855820 212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7474 prev_log_term 384 local_prev_log_term 387 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.857210 210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7473 prev_log_term 382 local_prev_log_term 385 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.859202 211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7472 prev_log_term 378 local_prev_log_term 383 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.860495 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7471 prev_log_term 376 local_prev_log_term 379 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.861850 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7470 prev_log_term 376 local_prev_log_term 377 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.863174 210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7469 prev_log_term 374 local_prev_log_term 375 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.864518 211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7468 prev_log_term 372 local_prev_log_term 373 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.865862 212 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7467 prev_log_term 370 local_prev_log_term 371 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.867173 207 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7466 prev_log_term 368 local_prev_log_term 369 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.868505 210 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7465 prev_log_term 366 local_prev_log_term 367 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
W20260305 08:51:32.869843 211 node.cpp:2462] node default_group:192.168.0.42:8107:8108 reject term_unmatched AppendEntries from 192.168.3.132:8107:8108 in term 410 prev_log_index 7464 prev_log_term 366 local_prev_log_term 365 last_log_index 7485 entries_size 0 from_append_entries_cache: 0
F20260305 08:51:32.872825 207 log_manager.cpp:310] Can't truncate logs before _applied_id=7485, last_log_kept=7463Olivier
03/05/2026, 8:56 AM*** Check failure stack trace: ***
E20260305 08:51:33.874399 207 backward.hpp:4200] Stack trace (most recent call last) in thread 207:
E20260305 08:51:33.874464 207 backward.hpp:4200] #19 Object "/opt/typesense-server", at 0x55cc4265ce40, in bthread_make_fcontext
E20260305 08:51:33.874476 207 backward.hpp:4200] #18 Object "/opt/typesense-server", at 0x55cc4267e2f8, in bthread::TaskGroup::task_runner(long)
E20260305 08:51:33.874485 207 backward.hpp:4200] #17 Object "/opt/typesense-server", at 0x55cc425c6794, in brpc::Socket::ProcessEvent(void*)
E20260305 08:51:33.874500 207 backward.hpp:4200] #16 Object "/opt/typesense-server", at 0x55cc4245920b, in brpc::InputMessenger::OnNewMessages(brpc::Socket*)
E20260305 08:51:33.874507 207 backward.hpp:4200] #15 Object "/opt/typesense-server", at 0x55cc4245831e, in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
E20260305 08:51:33.874516 207 backward.hpp:4200] #14 Object "/opt/typesense-server", at 0x55cc42458222, in brpc::ProcessInputMessage(void*)
E20260305 08:51:33.874523 207 backward.hpp:4200] #13 Object "/opt/typesense-server", at 0x55cc42467853, in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*)
E20260305 08:51:33.874532 207 backward.hpp:4200] #12 Object "/opt/typesense-server", at 0x55cc423fe1fa, in braft::RaftService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*)
E20260305 08:51:33.874540 207 backward.hpp:4200] #11 Object "/opt/typesense-server", at 0x55cc4239b989, in braft::RaftServiceImpl::append_entries(google::protobuf::RpcController*, braft::AppendEntriesRequest const*, braft::AppendEntriesResponse*, google::protobuf::Closure*)
E20260305 08:51:33.874547 207 backward.hpp:4200] #10 Object "/opt/typesense-server", at 0x55cc4231c71d, in braft::NodeImpl::handle_append_entries_request(brpc::Controller*, braft::AppendEntriesRequest const*, braft::AppendEntriesResponse*, google::protobuf::Closure*, bool)
E20260305 08:51:33.874555 207 backward.hpp:4200] #9 Object "/opt/typesense-server", at 0x55cc42343560, in braft::LogManager::append_entries(std::vector<braft::LogEntry*, std::allocatorbraft::LogEntry* >*, braft::LogManager::StableClosure*)
E20260305 08:51:33.874564 207 backward.hpp:4200] #8 Object "/opt/typesense-server", at 0x55cc4234321c, in braft::LogManager::check_and_resolve_conflict(std::vector<braft::LogEntry*, std::allocatorbraft::LogEntry* >*, braft::LogManager::StableClosure*)
E20260305 08:51:33.874572 207 backward.hpp:4200] #7 Object "/opt/typesense-server", at 0x55cc42342b9b, in braft::LogManager::unsafe_truncate_suffix(long)
E20260305 08:51:33.874583 207 backward.hpp:4200] #6 Object "/opt/typesense-server", at 0x55cc428f7529, in google::LogMessageFatal::~LogMessageFatal()
E20260305 08:51:33.874589 207 backward.hpp:4200] #5 Object "/opt/typesense-server", at 0x55cc428f3a5f, in google::LogMessage::Flush()
E20260305 08:51:33.874598 207 backward.hpp:4200] #4 Object "/opt/typesense-server", at 0x55cc428f422d, in google::LogMessage::SendToLog()
E20260305 08:51:33.874605 207 backward.hpp:4200] #3 Object "/opt/typesense-server", at 0x55cc428f42d1, in google::LogMessage::Fail()
E20260305 08:51:33.874616 207 backward.hpp:4200] #2 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb315bd57f2, in abort
E20260305 08:51:33.874624 207 backward.hpp:4200] #1 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb315bef475, in raise
E20260305 08:51:33.874632 207 backward.hpp:4200] #0 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fb315c439fc, in pthread_kill
Aborted (Signal sent by tkill() 1 10000)
I20260305 08:51:34.896282 207 housekeeper.cpp:96] No in-flight search queries were found.
E20260305 08:51:34.896332 207 typesense_server.cpp:159] Typesense 29.0 is terminating abruptly.Olivier
03/05/2026, 8:57 AMAlan Martini
03/05/2026, 3:31 PM