<@U01PL2YSG8L> we are facing cluster unhealthy <e...
# community-help
a
@Kishore Nallan we are facing cluster unhealthy ew5h2tr67livkp41p-1.a1.typesense.net
👀 1
a
Your cluster was completely threadpool exhausted. This means that a lot of imports likely timeouted, leaving threads allocated. You'll want to increase your write timeout in the client side to as high as 15 minutes. https://typesense.org/docs/guide/syncing-data-into-typesense.html#handling-http-503s
I've restarted it so it should be back in a few minutes
But you'll want to make this change as soon as possible
a
ok estimate time to back normal
a
5-40 minutes
Depends on your cluster size. And if this happens again due to the exact same problem, I will have to restart again
a
You'll want to increase your write timeout in the client side to as high as 15 minutes. how we can
a
It's a parameter in client initialization. You can try using the auto complete for checking it: https://github.com/typesense/typesense-python/blob/e863b4407ed6bc2635e8981b6552c1b4e2eff8ca/src/typesense/configuration.py#L70
Your cluster will be up in a few more minutes
nvm, it just got available
Btw, for the same amount of data, this is the time for initialization on your cluster. So whenever you restart it or we need to restart, it will take around 12 minutes
(unless ofc you add more data)
a
again unheathy we are implementing analytics event is this reason to unhealthy again and again
a
Dont think so. Same issue, threadpool exhaustion. Did you increase the timeout? You are having around 25rps for searches (around 10 more than usual)... THis could also mean your amount of CPU is just not enough to process the amount of requests.
Seems to have cleared. The fact that the threadpool keeps going up and down is another hint of just insufficient CPU