Typesense Server Crash and Counting Issue
TLDR Dan reported a server crash and counting issues with Typesense server version 0.26.0.rc1. Jason suggested an upgrade to 0.26.0.rc13
while Kishore Nallan committed to investigate the issue based on a sample dataset from Dan.
Aug 17, 2023 (1 month ago)
Dan
09:18 PMDan
09:18 PM
I20230817 17:00:51.051535 500549 raft_server.cpp:553] Term: 28, last_index index: 977955, committed_index: 977955, known_applied_index: 977955, applying_index:
0, queued_writes: 0, pending_queue_size: 0, local_sequence: 209268797
I20230817 17:00:51.051736 500620 raft_server.h:60] Peer refresh succeeded!
I20230817 17:01:01.053194 500549 raft_server.cpp:553] Term: 28, last_index index: 977955, committed_index: 977955, known_applied_index: 977955, applying_index:
0, queued_writes: 0, pending_queue_size: 0, local_sequence: 209268797
I20230817 17:01:01.053496 500616 raft_server.h:60] Peer refresh succeeded!
E20230817 17:01:07.575099 500294 backward.hpp:4200] Stack trace (most recent call last) in thread 500294:
E20230817 17:01:07.575278 500294 backward.hpp:4200] #13 Object "", at 0xffffffffffffffff, in
E20230817 17:01:07.575299 500294 backward.hpp:4200] #12 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc096b9e9ff, in
E20230817 17:01:07.575310 500294 backward.hpp:4200] #11 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc096b0cb42, in
E20230817 17:01:07.575343 500294 backward.hpp:4200] #10 Object "/usr/bin/typesense-server", at 0x167752d, in bvar::detail::SamplerCollector::sampling_thread(v
oid*)
E20230817 17:01:07.575361 500294 backward.hpp:4200] #9 Object "/usr/bin/typesense-server", at 0x1676e53, in bvar::detail::SamplerCollector::run()
E20230817 17:01:07.575379 500294 backward.hpp:4200] #8 Object "/usr/bin/typesense-server", at 0x1326e94, in bvar::detail::ReducerSampler<bvar::detail::Percen
tile, bvar::detail::PercentileSamples<254ul>, bvar::detail::Percentile::AddPercentileSamples, bvar::detail::VoidOp>::take_sample()
Log file created at: 2023/08/17 17:01:08
Running on machine: cst-prod-typesense-11
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230817 17:01:08.666965 662443 typesense_server_utils.cpp:329] Starting Typesense 0.26.0.rc1
I20230817 17:01:08.667083 662443 typesense_server_utils.cpp:332] Typesense is using jemalloc.
I20230817 17:01:08.667971 662443 typesense_server_utils.cpp:382] Thread pool size: 64
I20230817 17:01:08.677939 662443 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230817 17:01:09.159246 662443 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/meta
I20230817 17:01:09.183063 662443 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230817 17:01:09.183125 662443 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230817 17:01:09.183205 662443 typesense_server_utils.cpp:484] Starting API service...
I20230817 17:01:09.183355 662443 http_server.cpp:117] SSL cert refresh interval: 28800s
I20230817 17:01:09.183389 662699 typesense_server_utils.cpp:230] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20230817 17:01:09.183454 662700 batched_indexer.cpp:124] Starting batch indexer with 64 threads.
I20230817 17:01:09.186192 662700 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
I20230817 17:01:09.193367 662443 http_server.cpp:178] Typesense has started listening on port 443
Jason
09:37 PM0.26.0.rc13
?Dan
10:12 PMJason
10:24 PMJason
10:24 PMDan
11:08 PMAug 18, 2023 (1 month ago)
Kishore Nallan
12:41 AMKishore Nallan
12:42 AMDan
09:06 PMAug 19, 2023 (1 month ago)
Kishore Nallan
10:40 AMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.