What could be the reason to receive read timeouts ...
# community-helps
Stefan Hesse
05/27/2021, 2:55 PMWhat could be the reason to receive read timeouts on my cluster? Locally I can still reach it, also search works, but from my server I get read time out. I piped in a lot of data might there be some kind of AWS-ban that I ran into?
k
Kishore Nallan
05/27/2021, 2:59 PMAre you saying that the same call that succeeds on your local machine fails when run from your server?
s
Stefan Hesse
05/27/2021, 2:59 PMyes
Stefan Hesse
05/27/2021, 2:59 PMrequests.exceptions.ReadTimeout: HTTPSConnectionPool(host='6guhtikya0cf785np.a1.typesense.net', port=443): Read timed out. (read timeout=10)
k
Kishore Nallan
05/27/2021, 3:00 PMDoes a simple health check work? Hitting the /heath endpoint.
s
Stefan Hesse
05/27/2021, 3:01 PMStefan Hesse
05/27/2021, 3:01 PMlike that?
k
Kishore Nallan
05/27/2021, 3:02 PMYes, let me also take a look at metrics/logs on our end.
s
Stefan Hesse
05/27/2021, 3:02 PMcloud dashboard says all good
Stefan Hesse
05/27/2021, 3:03 PMStefan Hesse
05/27/2021, 3:03 PMdoes not succeed, neither locally nor on server
Stefan Hesse
05/27/2021, 3:05 PMokay I put http 😄
Stefan Hesse
05/27/2021, 3:05 PMlocally "ok" true, server timeout
k
Kishore Nallan
05/27/2021, 3:07 PMI think the
6guhtikya0cf785np-3Kishore Nallan
05/27/2021, 3:08 PMWhen you use the universal end-point with the "-x" it will try to hit the node that's closest to you.
Kishore Nallan
05/27/2021, 3:08 PMWhich might explain why it works locally but not from AWS.
Kishore Nallan
05/27/2021, 3:11 PMYup, that node is unhealthy. Thanks to HA your other two nodes will absorb the searches if the client is configured correctly. It will be replaced automatically if it does not recover soon.
s
Stefan Hesse
05/27/2021, 3:12 PMI just used this: "6guhtikya0cf785np.a1.typesense.net"
Stefan Hesse
05/27/2021, 3:12 PMendpoint, that's not enough for ha?
k
Kishore Nallan
05/27/2021, 3:13 PMNo, you have to give the full list of nodes. See the example here: https://typesense.org/docs/0.20.0/api/authentication.html#search-delivery-network
s
Stefan Hesse
05/27/2021, 3:13 PMah damn it
Stefan Hesse
05/27/2021, 3:13 PMokay thank you
Stefan Hesse
05/27/2021, 3:13 PMMaybe a note in the cloud central 😄
k
Kishore Nallan
05/27/2021, 3:14 PMSince for search delivery network we can't use a load balancer which cannot itself be distributed 🙂
Kishore Nallan
05/27/2021, 3:27 PM@Stefan Hesse Cluster has recovered now. https://6guhtikya0cf785np-3.a1.typesense.net/health is fine.
Kishore Nallan
05/27/2021, 3:28 PMYou might want to keep an eye on the free memory. Once you get within 50-75 MB on a small instance like this, anything can happen. I don't know if the previous node became unhealthy because of such an issue.
s
Stefan Hesse
05/27/2021, 3:28 PMgreat, thank you! Also upgraded my config
Stefan Hesse
05/27/2021, 3:31 PMcould be the case it seems to hold more memory now than it used to before with same amount of data, but I'd probably should upgrade to 1gb anyhow. Can you do that or should I create a new cluster?
k
Kishore Nallan
05/27/2021, 3:32 PMAs of now, create a new cluster. We're working on a round of much needed self serve improvements to the cloud UI.
2 Views