Hey guys I m getting some random crashes like this on my VMs typesense #community-help

Hey guys! I'm getting some random crashes like thi...

Matheus Bombonato

04/30/2025, 2:01 PM

Hey guys! I'm getting some random crashes like this on my VMs. I have 3 VMs, 32gb, 8 cores and good disks.

Copy code

E20250430 13:56:17.379438 2359282 backward.hpp:4200] #13   Object "", at 0xffffffffffffffff, in 
E20250430 13:56:17.379473 2359282 backward.hpp:4200] #12   Object "/usr/lib/x86_64-linux-gnu/libc-2.31.so", at 0x7f4d11730352, in __clone
E20250430 13:56:17.379501 2359282 backward.hpp:4200] #11   Object "/usr/lib/x86_64-linux-gnu/libpthread-2.31.so", at 0x7f4d1181b608, in start_thread
E20250430 13:56:17.379527 2359282 backward.hpp:4200] #10   Object "/usr/bin/typesense-server", at 0x5621640bcd23, in execute_native_thread_routine
E20250430 13:56:17.379554 2359282 backward.hpp:4200] #9  | Source "include/threadpool.h", line 59, in operator()
E20250430 13:56:17.379593 2359282 backward.hpp:4200]       Source "/usr/include/c++/10/future", line 1592, in ThreadPool [0x562161449d5c]
E20250430 13:56:17.379624 2359282 backward.hpp:4200] #8  | Source "/usr/include/c++/10/future", line 1459, in _M_set_result
E20250430 13:56:17.379654 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/future", line 412, in call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*>
E20250430 13:56:17.379685 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/mutex", line 729, in __gthread_once
E20250430 13:56:17.379714 2359282 backward.hpp:4200]       Source "/usr/include/x86_64-linux-gnu/c++/10/bits/gthr-default.h", line 700, in _M_run [0x562161668523]
E20250430 13:56:17.379766 2359282 backward.hpp:4200] #7    Object "/usr/lib/x86_64-linux-gnu/libpthread-2.31.so", at 0x7f4d118244de, in __pthread_once_slow
E20250430 13:56:17.379803 2359282 backward.hpp:4200] #6  | Source "/usr/include/c++/10/future", line 572, in operator()
E20250430 13:56:17.379832 2359282 backward.hpp:4200]       Source "/usr/include/c++/10/bits/std_function.h", line 622, in _M_do_set [0x562161448f72]
E20250430 13:56:17.379858 2359282 backward.hpp:4200] #5  | Source "/usr/include/c++/10/bits/std_function.h", line 292, in __invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run<std::_Bind<Index::batch_memory_index(Index*, std::vector<index_record>&, const string&, const tsl::htrie_map<char, field>&, const tsl::htrie_map<char, field>&, const string&, const std::vector<char>&, const std::vector<char>&, bool, size_t, size_t, size_t, bool, bool, const tsl::htrie_map<char, field>&, const string&, const spp::sparse_hash_map<std::__cxx11::basic_string<char>, std::set<reference_pair_t> >&)::<lambda()>()>, std::allocator<int>, void, {}>::<lambda()>, void>&>
E20250430 13:56:17.379886 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 115, in __invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<_Fn, _Alloc, _Res(_Args ...)>::_M_run<std::_Bind<Index::batch_memory_index(Index*, std::vector<index_record>&, const string&, const tsl::htrie_map<char, field>&, const tsl::htrie_map<char, field>&, const string&, const std::vector<char>&, const std::vector<char>&, bool, size_t, size_t, size_t, bool, bool, const tsl::htrie_map<char, field>&, const string&, const spp::sparse_hash_map<std::__cxx11::basic_string<char>, std::set<reference_pair_t> >&)::<lambda()>()>, std::allocator<int>, void, {}>::<lambda()>, void>&>
E20250430 13:56:17.379914 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()
E20250430 13:56:17.379962 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/future", line 1397, in operator()
E20250430 13:56:17.379993 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/future", line 1456, in __invoke_r<void, std::_Bind<Index::batch_memory_index(Index*, std::vector<index_record>&, const string&, const tsl::htrie_map<char, field>&, const tsl::htrie_map<char, field>&, const string&, const std::vector<char>&, const std::vector<char>&, bool, size_t, size_t, size_t, bool, bool, const tsl::htrie_map<char, field>&, const string&, const spp::sparse_hash_map<std::__cxx11::basic_string<char>, std::set<reference_pair_t> >&)::<lambda()>()>&>
E20250430 13:56:17.380020 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 110, in __invoke_impl<void, std::_Bind<Index::batch_memory_index(Index*, std::vector<index_record>&, const string&, const tsl::htrie_map<char, field>&, const tsl::htrie_map<char, field>&, const string&, const std::vector<char>&, const std::vector<char>&, bool, size_t, size_t, size_t, bool, bool, const tsl::htrie_map<char, field>&, const string&, const spp::sparse_hash_map<std::__cxx11::basic_string<char>, std::set<reference_pair_t> >&)::<lambda()>()>&>
E20250430 13:56:17.380046 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 60, in operator()<>
E20250430 13:56:17.380074 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/functional", line 499, in __call<void>
E20250430 13:56:17.380122 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/functional", line 416, in __invoke<Index::batch_memory_index(Index*, std::vector<index_record>&, const string&, const tsl::htrie_map<char, field>&, const tsl::htrie_map<char, field>&, const string&, const std::vector<char>&, const std::vector<char>&, bool, size_t, size_t, size_t, bool, bool, const tsl::htrie_map<char, field>&, const string&, const spp::sparse_hash_map<std::__cxx11::basic_string<char>, std::set<reference_pair_t> >&)::<lambda()>&>
E20250430 13:56:17.380167 2359282 backward.hpp:4200]     | Source "/usr/include/c++/10/bits/invoke.h", line 95, in __invoke_impl<void, Index::batch_memory_index(Index*, std::vector<index_record>&, const string&, const tsl::htrie_map<char, field>&, const tsl::htrie_map<char, field>&, const string&, const std::vector<char>&, const std::vector<char>&, bool, size_t, size_t, size_t, bool, bool, const tsl::htrie_map<char, field>&, const string&, const spp::sparse_hash_map<std::__cxx11::basic_string<char>, std::set<reference_pair_t> >&)::<lambda()>&>
E20250430 13:56:17.380198 2359282 backward.hpp:4200]       Source "/usr/include/c++/10/bits/invoke.h", line 60, in _M_invoke [0x5621616a58e6]
E20250430 13:56:17.380230 2359282 backward.hpp:4200] #4    Source "src/index.cpp", line 677, in operator() [0x5621616a55e2]
E20250430 13:56:17.380259 2359282 backward.hpp:4200] #3    Source "src/index.cpp", line 855, in index_field_in_memory [0x5621616a25df]
E20250430 13:56:17.380291 2359282 backward.hpp:4200] #2  | Source "src/facet_index.cpp", line 67, in emplace<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, facet_index_t::facet_id_seq_ids_t&>
E20250430 13:56:17.380326 2359282 backward.hpp:4200]       Source "/usr/include/c++/10/bits/stl_map.h", line 577, in insert [0x5621615e2069]
E20250430 13:56:17.380358 2359282 backward.hpp:4200] #1  | Source "/usr/include/c++/10/bits/stl_tree.h", line 2426, in _M_insert_node
E20250430 13:56:17.380388 2359282 backward.hpp:4200]       Source "/usr/include/c++/10/bits/stl_tree.h", line 2372, in _M_emplace_unique<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, facet_index_t::facet_id_seq_ids_t&> [0x5621615e58b8]
E20250430 13:56:17.380419 2359282 backward.hpp:4200] #0    Object "/usr/bin/typesense-server", at 0x56216404047a, in std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)

Matheus Bombonato

04/30/2025, 2:02 PM

Any ideas what this crash means? How to solve it? I'm running v28

Kishore Nallan

04/30/2025, 2:05 PM

Are you able to reproduce the issue?

Matheus Bombonato

04/30/2025, 2:05 PM

nope. No idea what's happening 😕

Kishore Nallan

04/30/2025, 2:08 PM

How big is your dataset

Matheus Bombonato

04/30/2025, 2:11 PM

about 20 indexes (one for each project). about 30gb in total. I'm refactoring it atm to use joins and reduce that.

Matheus Bombonato

04/30/2025, 2:11 PM

it's using about 20gb of memory

Kishore Nallan

04/30/2025, 2:12 PM

Disk size of the db?

Kishore Nallan

04/30/2025, 2:14 PM

Can you paste a few more log lines after the snippet you have given above?

Matheus Bombonato

04/30/2025, 2:14 PM

about 100gb

Kishore Nallan

04/30/2025, 2:14 PM

There should be something about saving current index that caused a crash

Matheus Bombonato

04/30/2025, 2:14 PM

this last crash happened when it was recovery from lag

Matheus Bombonato

04/30/2025, 2:16 PM

Copy code

50430 13:54:42.874249 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93576934, pending_writes: 0, queued_writes: 411459, local_sequence: 323070171
E20250430 13:54:50.875200 2359527 raft_server.cpp:758] 971065 lagging entries > healthy read lag of 1000
E20250430 13:54:50.875306 2359527 raft_server.cpp:770] 971065 lagging entries > healthy write lag of 500
I20250430 13:54:52.875735 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93576934, pending_writes: 0, queued_writes: 411459, local_sequence: 323070171
E20250430 13:54:59.876590 2359527 raft_server.cpp:758] 971065 lagging entries > healthy read lag of 1000
E20250430 13:54:59.876711 2359527 raft_server.cpp:770] 971065 lagging entries > healthy write lag of 500
I20250430 13:55:02.877153 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93576934, pending_writes: 0, queued_writes: 411459, local_sequence: 323070171
E20250430 13:55:08.877925 2359527 raft_server.cpp:758] 971065 lagging entries > healthy read lag of 1000
E20250430 13:55:08.878036 2359527 raft_server.cpp:770] 971065 lagging entries > healthy write lag of 500
I20250430 13:55:11.337677 2359528 batched_indexer.cpp:422] Running GC for aborted requests, req map size: 411461
I20250430 13:55:12.878603 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93577049, pending_writes: 0, queued_writes: 411471, local_sequence: 323070609
E20250430 13:55:14.697311 2359549 batched_indexer.cpp:257] Skipping write log index 93577676 which seems to have triggered a crash previously.
E20250430 13:55:17.879079 2359527 raft_server.cpp:758] 967051 lagging entries > healthy read lag of 1000
E20250430 13:55:17.879172 2359527 raft_server.cpp:770] 967051 lagging entries > healthy write lag of 500
I20250430 13:55:22.879782 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93585133, pending_writes: 0, queued_writes: 413364, local_sequence: 323098490
E20250430 13:55:26.880273 2359527 raft_server.cpp:758] 962239 lagging entries > healthy read lag of 1000
E20250430 13:55:26.880376 2359527 raft_server.cpp:770] 962239 lagging entries > healthy write lag of 500
I20250430 13:55:32.881019 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93589932, pending_writes: 0, queued_writes: 414368, local_sequence: 323115370
E20250430 13:55:35.881356 2359527 raft_server.cpp:758] 957480 lagging entries > healthy read lag of 1000
E20250430 13:55:35.881457 2359527 raft_server.cpp:770] 957480 lagging entries > healthy write lag of 500
I20250430 13:55:42.882222 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93591305, pending_writes: 0, queued_writes: 414586, local_sequence: 323120280
E20250430 13:55:44.882467 2359527 raft_server.cpp:758] 954484 lagging entries > healthy read lag of 1000
E20250430 13:55:44.882565 2359527 raft_server.cpp:770] 954484 lagging entries > healthy write lag of 500
I20250430 13:55:52.883356 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93595129, pending_writes: 0, queued_writes: 415303, local_sequence: 323133800
E20250430 13:55:53.883518 2359527 raft_server.cpp:758] 952789 lagging entries > healthy read lag of 1000
E20250430 13:55:53.883611 2359527 raft_server.cpp:770] 952789 lagging entries > healthy write lag of 500
I20250430 13:56:02.884524 2359527 raft_server.cpp:683] Term: 69, pending_queue: 0, last_index: 94547999, committed: 94547999, known_applied: 92793202, applying: 93600017, pending_writes: 0, queued_writes: 415893, local_sequence: 323152559
E20250430 13:56:02.884585 2359527 raft_server.cpp:758] 947982 lagging entries > healthy read lag of 1000
E20250430 13:56:02.884624 2359527 raft_server.cpp:770] 947982 lagging entries > healthy write lag of 500
E20250430 13:56:11.885402 2359527 raft_server.cpp:758] 945035 lagging entries > healthy read lag of 1000
E20250430 13:56:11.885507 2359527 raft_server.cpp:770] 945035 lagging entries > healthy write lag of 500

Matheus Bombonato

04/30/2025, 2:16 PM

I noticed this line:

Copy code

Skipping write log index 93577676 which seems to have triggered a crash previously.

Kishore Nallan

04/30/2025, 2:18 PM

Yes. It seems like some edge case in the data causing an issue during indexing of facet value.on that record.

Kishore Nallan

04/30/2025, 2:18 PM

When we detect a crash we try to skip that record on restart.

Kishore Nallan

04/30/2025, 2:19 PM

I wish there is some way to trigger this crash on a smaller dataset.

Open in Slack

Previous Next