Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16
TLDR Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.
1
Sep 21, 2023 (2 months ago)
Ankit
02:39 PMLet me know if you need any other info to reproduce/fix this one. Thanks!
Kishore Nallan
02:40 PM1
Sep 27, 2023 (2 months ago)
Ankit
11:12 AMI have used the schema and data from the github ticket.
CC: Jason Harpreet
Do let me know if you need anything else.
Jason
10:11 PMSep 28, 2023 (2 months ago)
Kishore Nallan
12:44 PM0.26.0.rc21
Ankit
01:55 PMThis maybe unrelated but I'm unable to upgrade my existing instance.
The log just keeps printing
I20230928 09:52:34.799630 53720 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I upgraded to rc21, it tried to load the index again but before it completed it just kept printing this log line. Downgrading to rc16 also just prints this log line. Tried this on 2 different instances. Both have enough RAM and CPU resources so it's not an OOM issue. Any ideas what I should be checking for?
Ankit
02:15 PMLog file created at: 2023/09/28 10:07:56
Running on machine: 534f46b42470
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230928 10:07:56.990743 54037 typesense_server_utils.cpp:332] Starting Typesense 0.26.0.rc21
I20230928 10:07:56.990875 54037 typesense_server_utils.cpp:335] Typesense is using jemalloc.
I20230928 10:07:56.991266 54037 typesense_server_utils.cpp:385] Thread pool size: 96
I20230928 10:07:57.003753 54037 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230928 10:07:57.012288 54037 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/meta
I20230928 10:07:57.023434 54037 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230928 10:07:57.023504 54037 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230928 10:07:57.023591 54037 typesense_server_utils.cpp:499] Starting API service...
I20230928 10:07:57.023917 54037 http_server.cpp:178] Typesense has started listening on port 8108
I20230928 10:07:57.023932 54389 typesense_server_utils.cpp:233] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20230928 10:07:57.024003 54390 batched_indexer.cpp:124] Starting batch indexer with 96 threads.
I20230928 10:07:57.028949 54390 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
I20230928 10:07:57.033095 54389 server.cpp:1107] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230928 10:07:57.033134 54389 server.cpp:1110] Check out in web browser.
I20230928 10:07:57.033362 54389 raft_server.cpp:68] Nodes configuration: 172.17.0.2:8107:8108
I20230928 10:07:57.033927 54389 log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I20230928 10:07:57.033989 54389 log.cpp:1172] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 444 time: 40
I20230928 10:07:57.034027 54389 log.cpp:1112] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20230928 10:07:57.048467 54435 raft_server.cpp:604] on_snapshot_load
I20230928 10:07:57.049295 54435 store.h:299] rm /var/lib/typesense/db success
I20230928 10:07:57.049527 54435 store.h:309] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000444/db_snapshot to /var/lib/typesense/db success
I20230928 10:07:57.049558 54435 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230928 10:07:57.063134 54435 store.h:323] DB open success!
I20230928 10:07:57.063169 54435 raft_server.cpp:583] Loading collections from disk...
I20230928 10:07:57.063189 54435 collection_manager.cpp:189] CollectionManager::load()
I20230928 10:07:57.063292 54435 auth_manager.cpp:34] Indexing 0 API key(s) found on disk.
I20230928 10:07:57.063342 54435 collection_manager.cpp:209] Loading upto 48 collections in parallel, 1000 documents at a time.
I20230928 10:07:57.063390 54435 collection_manager.cpp:218] Found 2 collection(s) on disk.
I20230928 10:07:57.065204 54517 collection_manager.cpp:139] Found collection int64_test with 4 memory shards.
I20230928 10:07:57.065375 54517 collection_manager.cpp:1575] Loading collection int64_test
I20230928 10:07:57.065830 54520 collection_manager.cpp:139] Found collection people_index_1695130274.3081167 with 4 memory shards.
I20230928 10:07:57.066123 54517 collection_manager.cpp:1693] Indexed 2/2 documents into collection int64_test
I20230928 10:07:57.066151 54517 collection_manager.cpp:274] Loaded 1 collection(s) so far
I20230928 10:07:57.066306 54520 collection_manager.cpp:1575] Loading collection people_index_1695130274.3081167
I20230928 10:08:58.036775 54390 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230928 10:09:59.041657 54390 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
The full log till the line keeps repeating every minute, if that helps.
Ankit
02:20 PMKishore Nallan
02:22 PMAnkit
02:23 PMAnkit
02:23 PMKishore Nallan
02:24 PMAnkit
02:25 PMKishore Nallan
02:25 PMAnkit
02:26 PMOct 02, 2023 (2 months ago)
Ankit
01:44 PMKishore Nallan
01:47 PMAnkit
01:48 PMKishore Nallan
01:50 PMTypesense
Indexed 3015 threads (79% resolved)
Similar Threads
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
Unresolved High-Volume Write Issue in Typesense
Greg experienced issues with Typesense where it became unresponsive during high-volume write operations. Jason and Kishore Nallan suggested several solutions, but the issue remained unresolved. They suspect that the problem occurs when concurrent writes are happening to the same collection.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Typesense Server Crash and Counting Issue
Dan reported a server crash and counting issues with Typesense server version 0.26.0.rc1. Jason suggested an upgrade to `0.26.0.rc13` while Kishore Nallan committed to investigate the issue based on a sample dataset from Dan.
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.