Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16
TLDR Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.
Powered by Struct AI
1
19
2mo
Sep 21, 2023 (2 months ago)
Ankit
Ankit
02:39 PMHey Kishore Nallan Jason Just reported this https://github.com/typesense/typesense/issues/1239
Let me know if you need any other info to reproduce/fix this one. Thanks!
Let me know if you need any other info to reproduce/fix this one. Thanks!
Kishore Nallan
Kishore Nallan
02:40 PMThanks, we will look immediately
1
Sep 27, 2023 (2 months ago)
Ankit
Ankit
11:12 AMRE: This issue, Here is the minimal app with Instant search. This renders the range filter with proper min and max values for 0.26rc1 and not when using 0.26rc16. Can't attach js/html in github.
I have used the schema and data from the github ticket.
CC: Jason Harpreet
Do let me know if you need anything else.
I have used the schema and data from the github ticket.
CC: Jason Harpreet
Do let me know if you need anything else.
Jason
Jason
10:11 PMThank you for that. Found the issue. We'll fix it shortly.
Sep 28, 2023 (2 months ago)
Kishore Nallan
Kishore Nallan
12:44 PMFixed in
0.26.0.rc21Ankit
Ankit
01:55 PMThanks!
This maybe unrelated but I'm unable to upgrade my existing instance.
The log just keeps printing
I upgraded to rc21, it tried to load the index again but before it completed it just kept printing this log line. Downgrading to rc16 also just prints this log line. Tried this on 2 different instances. Both have enough RAM and CPU resources so it's not an OOM issue. Any ideas what I should be checking for?
This maybe unrelated but I'm unable to upgrade my existing instance.
The log just keeps printing
I20230928 09:52:34.799630 53720 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0I upgraded to rc21, it tried to load the index again but before it completed it just kept printing this log line. Downgrading to rc16 also just prints this log line. Tried this on 2 different instances. Both have enough RAM and CPU resources so it's not an OOM issue. Any ideas what I should be checking for?
02:15
The full log till the line keeps repeating every minute, if that helps.
Ankit
02:15 PMLog file created at: 2023/09/28 10:07:56
Running on machine: 534f46b42470
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230928 10:07:56.990743 54037 typesense_server_utils.cpp:332] Starting Typesense 0.26.0.rc21
I20230928 10:07:56.990875 54037 typesense_server_utils.cpp:335] Typesense is using jemalloc.
I20230928 10:07:56.991266 54037 typesense_server_utils.cpp:385] Thread pool size: 96
I20230928 10:07:57.003753 54037 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230928 10:07:57.012288 54037 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/meta
I20230928 10:07:57.023434 54037 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230928 10:07:57.023504 54037 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230928 10:07:57.023591 54037 typesense_server_utils.cpp:499] Starting API service...
I20230928 10:07:57.023917 54037 http_server.cpp:178] Typesense has started listening on port 8108
I20230928 10:07:57.023932 54389 typesense_server_utils.cpp:233] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20230928 10:07:57.024003 54390 batched_indexer.cpp:124] Starting batch indexer with 96 threads.
I20230928 10:07:57.028949 54390 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
I20230928 10:07:57.033095 54389 server.cpp:1107] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230928 10:07:57.033134 54389 server.cpp:1110] Check out in web browser.
I20230928 10:07:57.033362 54389 raft_server.cpp:68] Nodes configuration: 172.17.0.2:8107:8108
I20230928 10:07:57.033927 54389 log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I20230928 10:07:57.033989 54389 log.cpp:1172] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 444 time: 40
I20230928 10:07:57.034027 54389 log.cpp:1112] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20230928 10:07:57.048467 54435 raft_server.cpp:604] on_snapshot_load
I20230928 10:07:57.049295 54435 store.h:299] rm /var/lib/typesense/db success
I20230928 10:07:57.049527 54435 store.h:309] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000444/db_snapshot to /var/lib/typesense/db success
I20230928 10:07:57.049558 54435 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230928 10:07:57.063134 54435 store.h:323] DB open success!
I20230928 10:07:57.063169 54435 raft_server.cpp:583] Loading collections from disk...
I20230928 10:07:57.063189 54435 collection_manager.cpp:189] CollectionManager::load()
I20230928 10:07:57.063292 54435 auth_manager.cpp:34] Indexing 0 API key(s) found on disk.
I20230928 10:07:57.063342 54435 collection_manager.cpp:209] Loading upto 48 collections in parallel, 1000 documents at a time.
I20230928 10:07:57.063390 54435 collection_manager.cpp:218] Found 2 collection(s) on disk.
I20230928 10:07:57.065204 54517 collection_manager.cpp:139] Found collection int64_test with 4 memory shards.
I20230928 10:07:57.065375 54517 collection_manager.cpp:1575] Loading collection int64_test
I20230928 10:07:57.065830 54520 collection_manager.cpp:139] Found collection people_index_1695130274.3081167 with 4 memory shards.
I20230928 10:07:57.066123 54517 collection_manager.cpp:1693] Indexed 2/2 documents into collection int64_test
I20230928 10:07:57.066151 54517 collection_manager.cpp:274] Loaded 1 collection(s) so far
I20230928 10:07:57.066306 54520 collection_manager.cpp:1575] Loading collection people_index_1695130274.3081167
I20230928 10:08:58.036775 54390 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230928 10:09:59.041657 54390 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 The full log till the line keeps repeating every minute, if that helps.
02:20
Ankit
02:20 PMFor some more context, this happened on an instance with 20mb data and with 6GB data.
Kishore Nallan
Kishore Nallan
02:22 PMThat's fine, it will be logged, it's just an info log
Ankit
Ankit
02:23 PMBut the re-indexing doesn't progress at all post this and doesn't finish for over 30 minutes.
02:23
Ankit
02:23 PMServer status is "Not ready or lagging" 503.
Kishore Nallan
Kishore Nallan
02:24 PMHmm, downgrading also doesn't work?
Ankit
Ankit
02:25 PMNo, downgrading didn't work.
Kishore Nallan
Kishore Nallan
02:25 PMWould you be able to zip the data directory of the smaller cluster and share via DM
Ankit
Ankit
02:26 PMYeah, give me a minute, will send that over.
Oct 02, 2023 (2 months ago)
Ankit
Ankit
01:44 PMHey, a question kind of related to this, do you use any threshold/sample data size to calculate the min and max value for an Int field? Or the whole data is being checked?
Kishore Nallan
Kishore Nallan
01:47 PMNo sampling is used unless you are sending facet_sample parameters in which case the min/max are calculated on the sampled records.
Ankit
Ankit
01:48 PMOkay. Thanks, we aren't using facet_sample, but seeing different results. Will share if I find a reproducable sample.
Kishore Nallan
Kishore Nallan
01:50 PMOk
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 3015 threads (79% resolved)
Similar Threads
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
31
7mo
Unresolved High-Volume Write Issue in Typesense
Greg experienced issues with Typesense where it became unresponsive during high-volume write operations. Jason and Kishore Nallan suggested several solutions, but the issue remained unresolved. They suspect that the problem occurs when concurrent writes are happening to the same collection.
43
3d
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
4
94
9mo
Typesense Server Crash and Counting Issue
Dan reported a server crash and counting issues with Typesense server version 0.26.0.rc1. Jason suggested an upgrade to `0.26.0.rc13` while Kishore Nallan committed to investigate the issue based on a sample dataset from Dan.
11
3mo
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.
57
13mo