#community-help

Typesense Cloud Instance Timeout Issue

TLDR Ken experienced timeouts during an upsert process. Kishore Nallan resolved the issue by fixing a disk problem and recommended upgrading to version 0.24.1.

Powered by Struct AI

1

1

21
4mo
Solved
Join the chat
May 16, 2023 (4 months ago)
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
12:39 PM
👋 Hello Typesense team! I’m looking for some support with out Typesense cloud instance ----->
12:39
Ken
12:39 PM
We have a process that runs every few hours that does a jsonl upsert of ~300k documents. It has been running fine for months, but as of today it started timing out.
12:39
Ken
12:39 PM
There hasn’t been any big change to our data or process that would explain this. We tested with a small number of documents (40) and it took a lot longer than expected. Wondering what your advice is to address this.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:39 PM
Please DM me your cluster ID
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
12:39 PM
My first thought was to drop/recreate the schema and re-run the upsert process (we do this periodically anyway when there has been a bigger change to our dataset). Any thoughts?
12:40
Ken
12:40 PM
done!
12:42
Ken
12:42 PM
I believe we are currently running 0.24.0.rcn28
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:43 PM
Checking

1

12:44
Kishore Nallan
12:44 PM
Somehow the write queue on this node is stuck. I can try restarting the Typesense process to see if it helps. It will require a few mins of downtime. Shall I proceed?
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
12:45 PM
yes, please go ahead
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:46 PM
Also, for what particular feature did you switch to the RC build?
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
12:47 PM
Trying to recall b/c it’s been a few months. I think it had to do with a newer filter or sort option.
12:48
Ken
12:48 PM
Like how null values are ranked in results
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:51 PM
Ah I see that your instance had a disk issue. This is now fixed and Typesense is starting back up.
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
12:53 PM
Great, thank you! I can see that our searches are running fine…
12:53
Ken
12:53 PM
I will try manually initiating the upsert
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:54 PM
I recommend upgrading to the last stable version 0.24.1 --> I think there is a fix for this particular issue that this build suffers from.
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
01:09 PM
OK, makes sense. I will run some local tests against the docker image for 0.24.1 and if everything looks good we’ll schedule an upgrade.
01:10
Ken
01:10 PM
BTW, our upsert completed fine in 123 seconds (consistent with past performance)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:10 PM
👍
Ken
Photo of md5-e647a31295419f1362630a69cf274ae3
Ken
01:11 PM
Thank you for the (as usual) excellent support!

1