Flushing Queue for Schema Altering in Typesense

Hi team! It looks like typesense have queue for schema altering, can I somehow flush it? I had bad configurator run and now I have Processing field additions and deletions first... every few minutes

Every few minutes? Does Typesense server keep getting restarted?

Right now not, but I have unlucky restarts + applying same drop + add field over and over again

There are no loops in the alter schema code, so if that keeps appearing then it probably means that the server is starting over and replaying recent writes from the write ahead log.

So I need just wait until all requests are done

> Running GC for aborted requests, req map size: 38
This is different, not related.

I’m using python client + self-written wrapper, which converts config file with schema declaration into actions which should make typesense state the same as. config. I had error in this code, so it had ran dropping + adding the same exact field over and over again. I’m just looking maybe I can speedup this process

> over and over again
AFAIK this should not happen at all because there is no inherent looping in the update schema code. So either of the two are happening:

a) Server is restarting after a crash and replaying recent writes from the write ahead log
b) The client is getting timed out and the official Typesense client retries the API call if there is a timeout

> This is different, not related.
Are you sure? It decreases every time after succesful processing:

Alter payload validation is successful...                                                                                                                                                              Processing field additions and deletions first...     
Running GC for aborted requests, req map size: 34                                                                                                                                                  Finished altering 105713 document(s).                                                                                                                                                                  Processing field modifications now...                                                                                                                                                                  
Finished altering 105713 document(s).


Alter payload validation is successful...                                                                                                                                                              
Processing field additions and deletions first...                                                                                                                                                      
Running GC for aborted requests, req map size: 33
Finished altering 105713 document(s).                                                                                                                                                                  
Processing field modifications now...
Finished altering 105713 document(s).

What that shows is that a bunch of requests have been queued up and they are being processed one by one. This happens for all writes.

So indeed the client seems to have sent a schema alter request repeatedly.

> a) Server is restarting after a crash and replaying recent writes from the write ahead log
This is exactly what happened

But the logs above don't suggest a restart. The requests are queued by and replayed.

Maybe because it was multiple time before and after restart? :thinking_face: My idea what was happened:
• I have statefulset with 1 node of typesense + post-update configurator job
• I ran heavy schema altering by error (delete two rows, index them again)
• It caused deploy timeout, so deploy was reverted (typesense restarted, post-update configurator job ran again)
• I retry this process a few times until found what’s going on

Yes that's possible

And yes, I got timeout in my client:
> HTTPConnectionPool(host=‘typesense’, port=1234): Read timed out. (read timeout=20)

Yes, definitely. Default value of a few seconds will be too small for larger collections. Or just use curl which does not timeout by default.

Yep, you also was right about python client. It looks like it retries requests even for patch/post requests

Which is fine for most cases, but maybe a gotcha for long running requests.

I will tweak timeout and max_retries