#community-help

Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16

TLDR Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.

Powered by Struct AI

1

Sep 21, 2023 (2 months ago)
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
02:39 PM
Hey Kishore Nallan Jason Just reported this https://github.com/typesense/typesense/issues/1239
Let me know if you need any other info to reproduce/fix this one. Thanks!
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:40 PM
Thanks, we will look immediately

1

Sep 27, 2023 (2 months ago)
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
11:12 AM
RE: This issue, Here is the minimal app with Instant search. This renders the range filter with proper min and max values for 0.26rc1 and not when using 0.26rc16. Can't attach js/html in github.
I have used the schema and data from the github ticket.
CC: Jason Harpreet

Do let me know if you need anything else.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
10:11 PM
Thank you for that. Found the issue. We'll fix it shortly.
Sep 28, 2023 (2 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:44 PM
Fixed in 0.26.0.rc21
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
01:55 PM
Thanks!
This maybe unrelated but I'm unable to upgrade my existing instance.
The log just keeps printing I20230928 09:52:34.799630 53720 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0

I upgraded to rc21, it tried to load the index again but before it completed it just kept printing this log line. Downgrading to rc16 also just prints this log line. Tried this on 2 different instances. Both have enough RAM and CPU resources so it's not an OOM issue. Any ideas what I should be checking for?
02:15
Ankit
02:15 PM
Log file created at: 2023/09/28 10:07:56
Running on machine: 534f46b42470
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20230928 10:07:56.990743 54037 typesense_server_utils.cpp:332] Starting Typesense 0.26.0.rc21
I20230928 10:07:56.990875 54037 typesense_server_utils.cpp:335] Typesense is using jemalloc.
I20230928 10:07:56.991266 54037 typesense_server_utils.cpp:385] Thread pool size: 96
I20230928 10:07:57.003753 54037 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230928 10:07:57.012288 54037 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/meta
I20230928 10:07:57.023434 54037 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230928 10:07:57.023504 54037 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230928 10:07:57.023591 54037 typesense_server_utils.cpp:499] Starting API service...
I20230928 10:07:57.023917 54037 http_server.cpp:178] Typesense has started listening on port 8108
I20230928 10:07:57.023932 54389 typesense_server_utils.cpp:233] Since no --nodes argument is provided, starting a single node Typesense cluster.
I20230928 10:07:57.024003 54390 batched_indexer.cpp:124] Starting batch indexer with 96 threads.
I20230928 10:07:57.028949 54390 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
I20230928 10:07:57.033095 54389 server.cpp:1107] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230928 10:07:57.033134 54389 server.cpp:1110] Check out  in web browser.
I20230928 10:07:57.033362 54389 raft_server.cpp:68] Nodes configuration: 172.17.0.2:8107:8108
I20230928 10:07:57.033927 54389 log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I20230928 10:07:57.033989 54389 log.cpp:1172] log load_meta /var/lib/typesense/state/log/log_meta first_log_index: 444 time: 40
I20230928 10:07:57.034027 54389 log.cpp:1112] load open segment, path: /var/lib/typesense/state/log first_index: 1
I20230928 10:07:57.048467 54435 raft_server.cpp:604] on_snapshot_load
I20230928 10:07:57.049295 54435 store.h:299] rm /var/lib/typesense/db success
I20230928 10:07:57.049527 54435 store.h:309] copy snapshot /var/lib/typesense/state/snapshot/snapshot_00000000000000000444/db_snapshot to /var/lib/typesense/db success
I20230928 10:07:57.049558 54435 store.h:64] Initializing DB by opening state dir: /var/lib/typesense/db
I20230928 10:07:57.063134 54435 store.h:323] DB open success!
I20230928 10:07:57.063169 54435 raft_server.cpp:583] Loading collections from disk...
I20230928 10:07:57.063189 54435 collection_manager.cpp:189] CollectionManager::load()
I20230928 10:07:57.063292 54435 auth_manager.cpp:34] Indexing 0 API key(s) found on disk.
I20230928 10:07:57.063342 54435 collection_manager.cpp:209] Loading upto 48 collections in parallel, 1000 documents at a time.
I20230928 10:07:57.063390 54435 collection_manager.cpp:218] Found 2 collection(s) on disk.
I20230928 10:07:57.065204 54517 collection_manager.cpp:139] Found collection int64_test with 4 memory shards.
I20230928 10:07:57.065375 54517 collection_manager.cpp:1575] Loading collection int64_test
I20230928 10:07:57.065830 54520 collection_manager.cpp:139] Found collection people_index_1695130274.3081167 with 4 memory shards.
I20230928 10:07:57.066123 54517 collection_manager.cpp:1693] Indexed 2/2 documents into collection int64_test
I20230928 10:07:57.066151 54517 collection_manager.cpp:274] Loaded 1 collection(s) so far
I20230928 10:07:57.066306 54520 collection_manager.cpp:1575] Loading collection people_index_1695130274.3081167
I20230928 10:08:58.036775 54390 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230928 10:09:59.041657 54390 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0

The full log till the line keeps repeating every minute, if that helps.
02:20
Ankit
02:20 PM
For some more context, this happened on an instance with 20mb data and with 6GB data.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:22 PM
That's fine, it will be logged, it's just an info log
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
02:23 PM
But the re-indexing doesn't progress at all post this and doesn't finish for over 30 minutes.
02:23
Ankit
02:23 PM
Server status is "Not ready or lagging" 503.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:24 PM
Hmm, downgrading also doesn't work?
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
02:25 PM
No, downgrading didn't work.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:25 PM
Would you be able to zip the data directory of the smaller cluster and share via DM
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
02:26 PM
Yeah, give me a minute, will send that over.
Oct 02, 2023 (2 months ago)
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
01:44 PM
Hey, a question kind of related to this, do you use any threshold/sample data size to calculate the min and max value for an Int field? Or the whole data is being checked?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:47 PM
No sampling is used unless you are sending facet_sample parameters in which case the min/max are calculated on the sampled records.
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
01:48 PM
Okay. Thanks, we aren't using facet_sample, but seeing different results. Will share if I find a reproducable sample.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:50 PM
Ok

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads