hi i get this error { \"message\": \"Not Ready or ...
# community-help
a
hi i get this error { \"message\": \"Not Ready or Lagging\"} when i try to delete my collections. Do you know what may cause this issue?
k
This indicates the the node is lagging behind in writes.
Check the
--healthy-read-lag
and
--healthy-write-lag
parameters here: https://typesense.org/docs/0.20.0/guide/configure-typesense.html#using-command-line-arguments
a
also, sometimes i get this message during import:
Post \"http://116.203.26.214:8108/collections\": read tcp 192.168.1.5053795 >116.203.26.2148108: wsarecv: An existing connection was forcibly closed by the remote host.
ok @Kishore Nallan, i'll check, thanks for the answer
k
I'm not sure about the other
forcibly closed
error. It indicates that the server has closed the connection with a
RST
packet, but it can also mean that some form of client / tcp timeout causing a connection drop. Finally (and least likely), check if the server itself crashed.
a
@Kishore Nallan server is alive, but still responds with {     "message": "Not Ready or Lagging" } for several minutes already
i tried restarting the service
k
Is this on Typesense cloud or self hosted?
a
selftr hosted
it just woke up
k
What do the logs say?
I think it was probably restarting and was catching up.
a
i'll try again and see if it repeats
hello @Kishore Nallan. I was able to reproduce the issue(Not Ready or Lagging) several times.
Right now the server doesnt allow me to upsert new batch of data
i saved several .log files from the several sessions where this problem occurred
could you please help me to review the log to identify the problem?
k
Okay, the server will be logging offsets -- if you can send a few lines of that, we can tell how big the lag is. But the problem is essentially about server lagging behind ingesting new data. I presume this is happening during updates, correct?
a
during upsert
typesense3.log
this is the full log
@Kishore Nallan could you pls let me know if you know the reason?
k
The writes are lagging behind, that is you are pushing upserts too fast and that is exceeding the write lag threshold configuration of 1000. See this snippet in the log line:
last_index index: 29278 ... applying_index: 27909
29278 - 27909 = 1369 entries.
a
hmm i see... Is it safe/ok to increase this threshold parameter?
k
To solve this, increase the
--healthy-read-lag
and
--healthy-write-lag
values to higher values. This way the server won't reject writes.
a
okay, i will try, thank you
k
It depends on your business use case. If a bit of lag makes no difference to your business, then it makes no difference.
It's basically a back pressure mechanism to warn you.
The upsert performance is going to be improved atleast 4-5x in the near future. I already have a build where it is much faster. It will be merged soon, and I can also share with you if you are interested once I finish benchmarking it further
If you run into this issue even after increasing threshold let me know, and we can try that build.
a
thank you!