How can i check cloud logs, were currently getting...
# community-helps
sam
05/26/2025, 10:33 AMHow can i check cloud logs, were currently getting constant unhealthy node2 and causing issues
sam
05/26/2025, 10:35 AMimage.png
f
Fanis Tharropoulos
05/26/2025, 10:39 AMThe metrics under it should show CPU usage and Ram Usage. Can you give us your Cluster's ID?
s
sam
05/26/2025, 10:40 AMaeqyxbt7ns6djrp0p
sam
05/26/2025, 10:40 AMjust upgrading away from burst cpu
f
Fanis Tharropoulos
05/26/2025, 10:42 AMThere's an upgrade request in progress. After it resolves, the node will be healthy again. Every time you upgrade / update the Typesense version, if you're not using High Availability, then there's going to be downtime like it is now
s
sam
05/26/2025, 10:45 AMwe have High Availability setup
sam
05/26/2025, 10:47 AMwe have Multi-Node HA configured
sam
05/26/2025, 10:47 AMdo you have any logs to show why it was timing out
sam
05/26/2025, 10:47 AMcause its impacting our ecom site at the minute
f
Fanis Tharropoulos
05/26/2025, 10:47 AMNodes are reporting healthy ATM
s
sam
05/26/2025, 10:48 AMYes, can you find the see the errors from the pervious two failures. seems to have happened in last hour
f
Fanis Tharropoulos
05/26/2025, 10:50 AMYour CPU spiked at 100% leading to unhealthy nodes. The upgrade process could initiate, since it requires nodes to be healthy, to not have any downtime
s
sam
05/26/2025, 10:52 AMWe did the upgrade after as we thought it might be because we was in burst mode and throttles to 15% after a while
sam
05/26/2025, 10:52 AMso just seeing if this stabilizes it
f
Fanis Tharropoulos
05/26/2025, 10:53 AMYour requests / s are about 1.5 -> 1.5 * 3600 * 24 * 30 -> 3.8 million requests / month. I suspect you need to upgrade to a 4vCPU instead of a 2vCPU configuration
s
sam
05/26/2025, 10:53 AMtripped to unhealthy again
sam
05/26/2025, 10:56 AMIs the load balancing working correctly, seems to be 1 node always spiking as well
sam
05/26/2025, 10:58 AMWe can look at increasing to 4vCPU but feels like its getting expensive quickly
f
Fanis Tharropoulos
05/26/2025, 10:59 AMAll three nodes spiked at 100%. If you want, we can upgrade you to 4vCPU. But there's added cost to that, so we need explicit access from you. Over the last 12 hours, your requests / s have spiked from an average of 0.6, to over 3.5. Have you just launched or are you experiencing bot traffic?
k
Kishore Nallan
05/26/2025, 11:24 AM@sam some of your queries are taking > 20 seconds to execute.
Kishore Nallan
05/26/2025, 11:26 AMDMing you a few sample slow queries that's showing up on our logs.
Kishore Nallan
05/26/2025, 11:29 AMCan you please try removing the
exhaustive_search=true