Ran into this crash while uploading synonyms on a ...
# community-help
j
Ran into this crash while uploading synonyms on a new cluster we just set up. We just created the collection and had not added any documents yet. It is the synonym list we uploaded successfully in another cluster.
a
Hey @Josh Handley Does it reliably reproduces this error whenever you import the synonym list? Also, could you kindly put the stack trace in a message inside the thread? (Like this one). Just so it doesn't make the channel flooded
j
Stack trace:
Copy code
W20251201 16:02:28.626052   595 node.cpp:2462] node default_group:10.4.50.7:8107:8108 reject term_unmatched AppendEntries from 10.4.38.7:8107:8108 in term 13 prev_log_index 1 prev_log_term 3 local_prev_log_term 2 last_log_index 6 entries_size 0 from_append_entries_cache: 0
F20251201 16:02:28.627214   487 log_manager.cpp:310] Can't truncate logs before _applied_id=6, last_log_kept=0
*** Check failure stack trace: ***
E20251201 16:02:30.201921   487 backward.hpp:4200] Stack trace (most recent call last) in thread 487:
E20251201 16:02:30.201954   487 backward.hpp:4200] #19   Object "/opt/typesense-server", at 0x5c3e61d4ce40, in bthread_make_fcontext
E20251201 16:02:30.201964   487 backward.hpp:4200] #18   Object "/opt/typesense-server", at 0x5c3e61d6e2f8, in bthread::TaskGroup::task_runner(long)
E20251201 16:02:30.201969   487 backward.hpp:4200] #17   Object "/opt/typesense-server", at 0x5c3e61cb6794, in brpc::Socket::ProcessEvent(void*)
E20251201 16:02:30.201975   487 backward.hpp:4200] #16   Object "/opt/typesense-server", at 0x5c3e61b4920b, in brpc::InputMessenger::OnNewMessages(brpc::Socket*)
E20251201 16:02:30.201980   487 backward.hpp:4200] #15   Object "/opt/typesense-server", at 0x5c3e61b4831e, in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure()
E20251201 16:02:30.201987   487 backward.hpp:4200] #14   Object "/opt/typesense-server", at 0x5c3e61b48222, in brpc::ProcessInputMessage(void*)
E20251201 16:02:30.201992   487 backward.hpp:4200] #13   Object "/opt/typesense-server", at 0x5c3e61b57853, in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*)
E20251201 16:02:30.202001   487 backward.hpp:4200] #12   Object "/opt/typesense-server", at 0x5c3e61aee1fa, in braft::RaftService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*)
E20251201 16:02:30.202006   487 backward.hpp:4200] #11   Object "/opt/typesense-server", at 0x5c3e61a8b989, in braft::RaftServiceImpl::append_entries(google::protobuf::RpcController*, braft::AppendEntriesRequest const*, braft::AppendEntriesResponse*, google::protobuf::Closure*)
E20251201 16:02:30.202011   487 backward.hpp:4200] #10   Object "/opt/typesense-server", at 0x5c3e61a0c71d, in braft::NodeImpl::handle_append_entries_request(brpc::Controller*, braft::AppendEntriesRequest const*, braft::AppendEntriesResponse*, google::protobuf::Closure*, bool)
E20251201 16:02:30.202018   487 backward.hpp:4200] #9    Object "/opt/typesense-server", at 0x5c3e61a33560, in braft::LogManager::append_entries(std::vector<braft::LogEntry*, std::allocator<braft::LogEntry*> >*, braft::LogManager::StableClosure*)
E20251201 16:02:30.202023   487 backward.hpp:4200] #8    Object "/opt/typesense-server", at 0x5c3e61a3321c, in braft::LogManager::check_and_resolve_conflict(std::vector<braft::LogEntry*, std::allocator<braft::LogEntry*> >*, braft::LogManager::StableClosure*)
E20251201 16:02:30.202029   487 backward.hpp:4200] #7    Object "/opt/typesense-server", at 0x5c3e61a32b9b, in braft::LogManager::unsafe_truncate_suffix(long)
E20251201 16:02:30.202035   487 backward.hpp:4200] #6    Object "/opt/typesense-server", at 0x5c3e61fe7529, in google::LogMessageFatal::~LogMessageFatal()
E20251201 16:02:30.202041   487 backward.hpp:4200] #5    Object "/opt/typesense-server", at 0x5c3e61fe3a5f, in google::LogMessage::Flush()
E20251201 16:02:30.202047   487 backward.hpp:4200] #4    Object "/opt/typesense-server", at 0x5c3e61fe422d, in google::LogMessage::SendToLog()
E20251201 16:02:30.202055   487 backward.hpp:4200] #3    Object "/opt/typesense-server", at 0x5c3e61fe42d1, in google::LogMessage::Fail()
E20251201 16:02:30.202065   487 backward.hpp:4200] #2    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f5d15da47f2, in abort
E20251201 16:02:30.202072   487 backward.hpp:4200] #1    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f5d15dbe475, in raise
E20251201 16:02:30.202078   487 backward.hpp:4200] #0    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f5d15e129fc, in pthread_kill
Aborted (Signal sent by tkill() 1 10000)
I20251201 16:02:31.796948   487 housekeeper.cpp:96] No in-flight search queries were found.
E20251201 16:02:31.797006   487 typesense_server.cpp:159] Typesense 29.0 is terminating abruptly.
The pod that hit this crash is in a crash loop backoff continually hitting this same error. Let me see if I can recover it and try importing the synonyms again.
a
By the stacktrace alone, it seems like an issue in the Logging lib we use, rather than synonyms. If you can create a reproducible example that would be great. We have this gist as a good starting point: https://gist.github.com/auth/github?return_to=https%3A%2F%2Fgist.github.com%2Fjasonbosco%2F7c3432713216c378472f13e72246f46b
j
Unfortunately I haven't been able to reproduce. I deleted the pod and associated PV per this issue and now it seems to be happily uploading synonyms again.