#community-help

Typesense Server Crash and Counting Issue

TLDR Dan reported a server crash and counting issues with Typesense server version 0.26.0.rc1. Jason suggested an upgrade to 0.26.0.rc13 while Kishore Nallan committed to investigate the issue based on a sample dataset from Dan.

3mo

Join the chat

Aug 17, 2023 (3 months ago)

Photo of md5-5f114bcaa05d5c832038d78174f2060d

Dan

09:18 PM

Hi typesense team - we ran into this issue with our typesense server crashing and restarting. See the log from the server in thread. I think we’re on 0.26.0.rc1. Any idea what could cause this?

09:18

Dan

09:18 PM


I20230817 17:00:51.051535 500549 raft_server.cpp:553] Term: 28, last_index index: 977955, committed_index: 977955, known_applied_index: 977955, applying_index: 
0, queued_writes: 0, pending_queue_size: 0, local_sequence: 209268797
I20230817 17:00:51.051736 500620 raft_server.h:60] Peer refresh succeeded!
I20230817 17:01:01.053194 500549 raft_server.cpp:553] Term: 28, last_index index: 977955, committed_index: 977955, known_applied_index: 977955, applying_index: 
0, queued_writes: 0, pending_queue_size: 0, local_sequence: 209268797
I20230817 17:01:01.053496 500616 raft_server.h:60] Peer refresh succeeded!
E20230817 17:01:07.575099 500294 backward.hpp:4200] Stack trace (most recent call last) in thread 500294:
E20230817 17:01:07.575278 500294 backward.hpp:4200] #13   Object "", at 0xffffffffffffffff, in 
E20230817 17:01:07.575299 500294 backward.hpp:4200] #12   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc096b9e9ff, in 
E20230817 17:01:07.575310 500294 backward.hpp:4200] #11   Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc096b0cb42, in 
E20230817 17:01:07.575343 500294 backward.hpp:4200] #10   Object "/usr/bin/typesense-server", at 0x167752d, in bvar::detail::SamplerCollector::sampling_thread(v
oid*)
E20230817 17:01:07.575361 500294 backward.hpp:4200] #9    Object "/usr/bin/typesense-server", at 0x1676e53, in bvar::detail::SamplerCollector::run()
E20230817 17:01:07.575379 500294 backward.hpp:4200] #8    Object "/usr/bin/typesense-server", at 0x1326e94, in bvar::detail::ReducerSampler<bvar::detail::Percen
tile, bvar::detail::PercentileSamples<254ul>, bvar::detail::Percentile::AddPercentileSamples, bvar::detail::VoidOp>::take_sample()
Log file created at: 2023/08/17 17:01:08
Running on machine: cst-prod-typesense-11
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230817 17:01:08.666965 662443 typesense_server_utils.cpp:329] Starting Typesense 0.26.0.rc1
I20230817 17:01:08.667083 662443 typesense_server_utils.cpp:332] Typesense is using jemalloc.
I20230817 17:01:08.667971 662443 typesense_server_utils.cpp:382] Thread pool size: 64
I20230817 17:01:08.677939 662443 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230817 17:01:09.159246 662443 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/meta
I20230817 17:01:09.183063 662443 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230817 17:01:09.183125 662443 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230817 17:01:09.183205 662443 typesense_server_utils.cpp:484] Starting API service...
I20230817 17:01:09.183355 662443 http_server.cpp:117] SSL cert refresh interval: 28800s
I20230817 17:01:09.183389 662699 typesense_server_utils.cpp:230] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20230817 17:01:09.183454 662700 batched_indexer.cpp:124] Starting batch indexer with 64 threads.
I20230817 17:01:09.186192 662700 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
I20230817 17:01:09.193367 662443 http_server.cpp:178] Typesense has started listening on port 443

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

09:37 PM

Could you try upgrading to 0.26.0.rc13?

Dan

10:12 PM

yeah we can try it. is there a place with release notes where we can see what’s included or changed. we were waiting on some facet search optimizations too (my collegue Ankit has been in touch with your team). i’m curious if those changes are in rc13

Jason

10:24 PM

There are a few improvements, but we’re continuing to iterate on it

10:24

Jason

10:24 PM

We don’t publish release notes for each RC build…

Dan

11:08 PM

Jason just tried the 0.26rc13, the most notable change is that the facet counts are incorrectly and much higher than what they should be. we currently have ~700k records in the collection but some facets are show counts in millions

Aug 18, 2023 (3 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

12:41 AM

Hi Dan, it must be a regression. Once again, I request you to share a sample dataset that we can reproduce and investigate.

12:42

Kishore Nallan

12:42 AM

I'll also try with our datasets to see if we missed something in this latest build.

Dan

09:06 PM

Kishore Nallan i’ll DM you the sample dataset, thanks for your help!

Aug 19, 2023 (3 months ago)

Kishore Nallan

10:40 AM

Thank you, will check and get back to you early next week.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16

Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.

2mo

Typesense Node Stuck in Segfault Loop After Stress Test

Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.

7mo

Large JSONL Documents Import Issue & Resolution

Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.

9mo

Unresolved High-Volume Write Issue in Typesense

Greg experienced issues with Typesense where it became unresponsive during high-volume write operations. Jason and Kishore Nallan suggested several solutions, but the issue remained unresolved. They suspect that the problem occurs when concurrent writes are happening to the same collection.

Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

13mo