#community-help

Flushing Queue for Schema Altering in Typesense

TLDR Dima experienced repeated schema alterations due to a misconfiguration, causing server timeouts. Kishore Nallan recommended increasing the timeout and adjusting max_retries to resolve the issue.

Powered by Struct AI

3

28
6mo
Solved
Join the chat
Mar 28, 2023 (6 months ago)
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:29 AM
Hi team! It looks like typesense have queue for schema altering, can I somehow flush it? I had bad configurator run and now I have Processing field additions and deletions first... every few minutes
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:31 AM
Every few minutes? Does Typesense server keep getting restarted?
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:31 AM
Right now not, but I have unlucky restarts + applying same drop + add field over and over again
10:31
Dima
10:31 AM
Running GC for aborted requests, req map size: 38
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:31 AM
There are no loops in the alter schema code, so if that keeps appearing then it probably means that the server is starting over and replaying recent writes from the write ahead log.
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:32 AM
So I need just wait until all requests are done
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:32 AM
> Running GC for aborted requests, req map size: 38
This is different, not related.
10:32
Kishore Nallan
10:32 AM
How are you applying the schema update? What client do yo uuse?
10:33
Kishore Nallan
10:33 AM
I think what could be happening is that the operation takes a long time, but your client has a short default timeout and it gets retried over and over again.
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:35 AM
I’m using python client + self-written wrapper, which converts config file with schema declaration into actions which should make typesense state the same as. config. I had error in this code, so it had ran dropping + adding the same exact field over and over again. I’m just looking maybe I can speedup this process
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:37 AM
> over and over again
AFAIK this should not happen at all because there is no inherent looping in the update schema code. So either of the two are happening:

a) Server is restarting after a crash and replaying recent writes from the write ahead log
b) The client is getting timed out and the official Typesense client retries the API call if there is a timeout
10:37
Kishore Nallan
10:37 AM
I will rule out b) first. Ensure that the client has a large timeout value configured so that the write call is not retried on timeout error.
10:38
Kishore Nallan
10:38 AM
The other thing that could have happened is that the client treats the bad schema request as a server error and retries :thinking_face:
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:39 AM
> This is different, not related.
Are you sure? It decreases every time after succesful processing:
Alter payload validation is successful...                                                                                                                                                              Processing field additions and deletions first...     
Running GC for aborted requests, req map size: 34                                                                                                                                                  Finished altering 105713 document(s).                                                                                                                                                                  Processing field modifications now...                                                                                                                                                                  
Finished altering 105713 document(s).


Alter payload validation is successful...                                                                                                                                                              
Processing field additions and deletions first...                                                                                                                                                      
Running GC for aborted requests, req map size: 33
Finished altering 105713 document(s).                                                                                                                                                                  
Processing field modifications now...
Finished altering 105713 document(s).
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:39 AM
What that shows is that a bunch of requests have been queued up and they are being processed one by one. This happens for all writes.

So indeed the client seems to have sent a schema alter request repeatedly.

1

10:40
Kishore Nallan
10:40 AM
One way to stop this is to restart the server with the --skip-writes flag.
10:41
Kishore Nallan
10:41 AM
This will skip ALL queued writes. Then you can trigger a manual snapshot call (check operations end-point) and then those logs will be purged and then you are restart server again without that flag.
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:41 AM
> a) Server is restarting after a crash and replaying recent writes from the write ahead log
This is exactly what happened
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:42 AM
But the logs above don't suggest a restart. The requests are queued by and replayed.
10:42
Kishore Nallan
10:42 AM
Unless you see a stack trace along with the crash?
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:48 AM
Maybe because it was multiple time before and after restart? :thinking_face: My idea what was happened:
• I have statefulset with 1 node of typesense + post-update configurator job
• I ran heavy schema altering by error (delete two rows, index them again)
• It caused deploy timeout, so deploy was reverted (typesense restarted, post-update configurator job ran again)
• I retry this process a few times until found what’s going on
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:51 AM
Yes that's possible
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:53 AM
And yes, I got timeout in my client:
> HTTPConnectionPool(host=‘typesense’, port=1234): Read timed out. (read timeout=20)
10:53
Dima
10:53 AM
Do I need to set up bigger timeout for altering field schema migration?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:54 AM
Yes, definitely. Default value of a few seconds will be too small for larger collections. Or just use curl which does not timeout by default.
Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:58 AM
Yep, you also was right about python client. It looks like it retries requests even for patch/post requests
Image 1 for Yep, you also was right about python client. It looks like it retries requests even for patch/post requests
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:59 AM
Which is fine for most cases, but maybe a gotcha for long running requests.

1

Dima
Photo of md5-1b62114a658b760944aa7d2b4c274460
Dima
10:59 AM
I will tweak timeout and max_retries

1