Typesense Server Crash and Counting Issue
TLDR Dan reported a server crash and counting issues with Typesense server version 0.26.0.rc1. Jason suggested an upgrade to 0.26.0.rc13
while Kishore Nallan committed to investigate the issue based on a sample dataset from Dan.
Aug 17, 2023 (3 months ago)
Dan
09:18 PMDan
09:18 PM
I20230817 17:00:51.051535 500549 raft_server.cpp:553] Term: 28, last_index index: 977955, committed_index: 977955, known_applied_index: 977955, applying_index:
0, queued_writes: 0, pending_queue_size: 0, local_sequence: 209268797
I20230817 17:00:51.051736 500620 raft_server.h:60] Peer refresh succeeded!
I20230817 17:01:01.053194 500549 raft_server.cpp:553] Term: 28, last_index index: 977955, committed_index: 977955, known_applied_index: 977955, applying_index:
0, queued_writes: 0, pending_queue_size: 0, local_sequence: 209268797
I20230817 17:01:01.053496 500616 raft_server.h:60] Peer refresh succeeded!
E20230817 17:01:07.575099 500294 backward.hpp:4200] Stack trace (most recent call last) in thread 500294:
E20230817 17:01:07.575278 500294 backward.hpp:4200] #13 Object "", at 0xffffffffffffffff, in
E20230817 17:01:07.575299 500294 backward.hpp:4200] #12 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc096b9e9ff, in
E20230817 17:01:07.575310 500294 backward.hpp:4200] #11 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc096b0cb42, in
E20230817 17:01:07.575343 500294 backward.hpp:4200] #10 Object "/usr/bin/typesense-server", at 0x167752d, in bvar::detail::SamplerCollector::sampling_thread(v
oid*)
E20230817 17:01:07.575361 500294 backward.hpp:4200] #9 Object "/usr/bin/typesense-server", at 0x1676e53, in bvar::detail::SamplerCollector::run()
E20230817 17:01:07.575379 500294 backward.hpp:4200] #8 Object "/usr/bin/typesense-server", at 0x1326e94, in bvar::detail::ReducerSampler<bvar::detail::Percen
tile, bvar::detail::PercentileSamples<254ul>, bvar::detail::Percentile::AddPercentileSamples, bvar::detail::VoidOp>::take_sample()
Log file created at: 2023/08/17 17:01:08
Running on machine: cst-prod-typesense-11
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230817 17:01:08.666965 662443 typesense_server_utils.cpp:329] Starting Typesense 0.26.0.rc1
I20230817 17:01:08.667083 662443 typesense_server_utils.cpp:332] Typesense is using jemalloc.
I20230817 17:01:08.667971 662443 typesense_server_utils.cpp:382] Thread pool size: 64
I20230817 17:01:08.677939 662443 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230817 17:01:09.159246 662443 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/meta
I20230817 17:01:09.183063 662443 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230817 17:01:09.183125 662443 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230817 17:01:09.183205 662443 typesense_server_utils.cpp:484] Starting API service...
I20230817 17:01:09.183355 662443 http_server.cpp:117] SSL cert refresh interval: 28800s
I20230817 17:01:09.183389 662699 typesense_server_utils.cpp:230] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20230817 17:01:09.183454 662700 batched_indexer.cpp:124] Starting batch indexer with 64 threads.
I20230817 17:01:09.186192 662700 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
I20230817 17:01:09.193367 662443 http_server.cpp:178] Typesense has started listening on port 443
Jason
09:37 PM0.26.0.rc13
?Dan
10:12 PMJason
10:24 PMJason
10:24 PMDan
11:08 PMAug 18, 2023 (3 months ago)
Kishore Nallan
12:41 AMKishore Nallan
12:42 AMDan
09:06 PMAug 19, 2023 (3 months ago)
Kishore Nallan
10:40 AMTypesense
Indexed 3015 threads (79% resolved)
Similar Threads
Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16
Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Unresolved High-Volume Write Issue in Typesense
Greg experienced issues with Typesense where it became unresponsive during high-volume write operations. Jason and Kishore Nallan suggested several solutions, but the issue remained unresolved. They suspect that the problem occurs when concurrent writes are happening to the same collection.
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.